An insider in the New York City Department of Education was disturbed to read the New York Times’ editorial praising the CREDO study of charters in New York City. She knew that the data on the public website of the Department of Education does not support the CREDO analysis.
Here is her own analysis, based on DOE’s own data:
A New York Times editorial Saturday praised a new study claiming that through the 2010-11 school year New York City charter schools have produced better results for students than other public schools. Of course this study did not mention the investigative reporting by Reuters proving that charter schools have truckloads of schemes to turn away and kick out students who might bring down their numbers. See that story here.
Nonetheless we took the report on good faith and attempted to verify its claims using the New York City Department of Education’s own data sets that can be found here: http://schools.nyc.gov/Accountability/tools/report/default.htm In these data sets charter school outcomes are compared to those of similar schools. Similar schools are schools that educate comparable students based on incoming test scores and other criteria. We decided to spend some time examining these data sets for the 2009-10 and 2010-11 school years. There are the years in which the NY Times claimed charter schools had “whopping” results. But is it true? Sadly the NY Times has been fooled.
Let’s start with elementary and middle charter schools during the 2009-10 school year. In that year the average student results on the English exam for charter schools in New York City placed them at the 32.5th percentile of similar schools. Looking at students who scored as “proficient” on the English exam charter schools were at the 31.4th percentile of similar schools. And when looking at how charter schools helped their students improve as compared to prior years (in other words: student growth) they performed dismally, ranking at the 20.1st percentile.
But how about math? Maybe charters don’t do such a good job with English but do a better job in math. Well they do, but still performed worse than about 60% of public schools with similar student populations: ranking at the 42.3rd percentile for average student test scores, the 37.9th percentile for students scoring “proficient” on math, and coming in at the 44.8th percentile for student improvement in math.
In 2010-11, the school year in which the study cited by the New York Times claimed the best performance by charter schools, the truth is that charter schools continued to do a much, much poorer job for students than other schools with similar students. In English they ranked at the 35.1st percentile for average student test scores, at the 36.7th percentile for students scoring “proficient” and at the lowly 28.6th percentile for student growth. Math was slightly better: 51.5th percentile for average student scores, 55.4th for student “proficiency,” and 52.2nd for student growth. And this is with the charter school practice of kicking out students right before testing time as shown in this expose http://www.edwize.org/middle-school-charters-show-alarming-student-attrition
So where does this leave us? Charter schools, in fact, did much, much worse than similar schools in NYC in English and about average in math. If you average the numbers for 2010-11 together, charter schools are doing 16.5% worse than the average similar school in English. And, using the same calculation, they do 3% better than the average similar school in Math. It would appear that much, much poorer performance in English and barely better performance in Math does not support the claim that charter schools in New York City give superior results, whatever the New York Times might say. It should also be pointed out that these comparisons are based on ACTUAL students unlike the study cited so approvingly by the New York Times, which invented virtual students to do their “comparisons.”
How about high schools? Maybe charter high schools in New York City are doing a “whopping” job there. Again, unfortunately for students, they do not. Using the data set available on the New York City Department of Education webpage here: http://schools.nyc.gov/NR/rdonlyres/C8903442-BA48-4248-B4CE-156FAA2D8929/0/2010_2011_HS_PR_Results_2012_03_16.xlsx we find that in 2010-11, for the first time, schools were rated based on how well they were preparing students for college. And how did charter high schools do on those ratings? Only 14.9 percent of the charter high school students met the college ready standard as compared to a 32.3 percent average in similar high schools. Only 31.4 percent of charter high school students took and passed a college preparatory course as compared to a 42.8 percent average in similar high schools. And only 52.8 percent of students enrolled in college as compared to a 61.2 percent average in similar schools.
What does this all mean for education? We must start to evaluate the success and failures of initiatives such as charter schools truthfully without letting politics get in the way. We owe this to students. Unfortunately, the response from charter schools, their protectors and funders will probably be a redoubled effort to screen and selectively prune students at charter schools to make their numbers look better. They will continue, with the support of the editorial page of the New York Times, to bash public schools. Instead of committing to improving education for all students and giving all schools the resources to do what we know helps all kids (strong curriculum, small classes, a pleasant school environment, high quality after-school programs, embedded systems of social-emotional and health supports) they will continue to play politics with public education and the futures of our children.
What’s CREDO? What does its initials stand for?
Center for Research on EDucation Outcomes. It’s affiliated with Stanford.
CREDO is affiliated with Stanford. Macke Raymond is affiliated with Hoover Institution. CREDO funded by Walton Foundation.
Facts, not spin, are amazing things and information. The ideologists always claim that a failure is a success. Look at the Gates Foundation and their over $1 billion over 10 years on small schools. For 10 years this was the only thing that mattered then suddenly they discovered that it was a failure and Tom Van Der Ark goes to N.Y. to start 3 charters and fails and leaves an over $1 million debt and one week after John Deasy quits his job as superintendent of Prince Georges County because of the stories on his phony PHD. Suddenly Gates is all about taking down teachers once again without any evidence and no apology about their lack of understanding the faulty “Small Schools.” About 2 years after that Deasy is superintendent of LAUSD bringing more destruction to LAUSD after the major destruction of Roy Romer. LAUSD has a general fund budget of $5.8 billion. N.Y. has $23.9 billion according to the latest Deloitte and Touche audit. Isn’t it interesting that N.Y. told parents it was about $3 billion. That is close isn’t it? Real Pros.
Do you mean to say that NYC told parents that the DOE’s education budget was $3 billion? That would be so patently ridiculous that frankly I can’t believe it happened.
George, the Gates Foundation spent $2 billion on small schools, not $1 billion.
I’m not quite clear on what I’m supposed to conclude about this statement from the Times story:
Is the point that this “DOE insider” couldn’t find any of these numbers (or numbers to support those numbers) in the CREDO study, and that the implication is that the numbers are made up? Or are these numbers in the report but misinterpreted by the Times? If the latter, what’s the misinterpretation?
They always make things up to support their failures. Like LAUSD in 2010-11 stating in their superintendents budget ADA numbers that are 72,000 different than those on the CDE website. How does this happen? Or Superintendent Deasy and Board President Garcia testifying that LAUSD had only $4,800/student when it was really $11,233 according to their own budget and the CDE website. How does this happen? By accident or they did not know? Do you really believe that? If true then both have no business in those positions. For over 20 years we have discovered dual or more reporting of different numbers to different agencies as they know that these agencies do not talk to each other so unless someone does an outside investigation no one knows. This is normal when big money is involved.
The Times took the figures straight from the CREDO study. So the assertion is, ultimately, that the CREDO numbers don’t gel with the DOE’s numbers. Which numbers are better? Heck if I know. But I would advise anybody who’s been trumpeting CREDO study results that support their arguments to think twice about whether they want to deride the new study’s data, since the methodology appears to have been the same for all the studies.
Here’s an excerpt from the CREDO study, from its methodology description:
So is the “DOE insider” critique based on the premise that the CREDO methodology is invalid, and the study’s data unreliable? I never heard a peep about this from people who cited CREDO’s study as evidence that most charter schools were worse or no better than regular public schools.
The CREDO study creates virtual matches. The insider uses DOE data.
I assume you don’t have a position one way or the other on whether the DOE data is more accurate than the “virtual matches” used in the CREDO study.
This is exactly the data needed to understand what is happening. I was at a meeting at LAUSD today where there was a presentation on statistics by a professor from another state. I could not believe that they are such ignorant people on statistics and have not used them when it is so easy in California with the CDE website and the districts access to their own data. Why did they have to go 1/2 way across the country for such simple data? Do they mean to say that there is no one at LAUSD who knows or gathers and analyzes data? Apparently so. I have been doing this for years and far beyond what they are talking about and have been presenting it for years. And they are now just waking up to it. We even had a UTLA representative to this committee deny her affiliation with UTLA and would not answer who she represented even when it is printed on the agenda and on her sign in front of her on the dais. Is this acceptable? Does UTLA really think this will gain them favor? The problem with her was her statement concerning the fact of whether LAUSD’s revenue is the same as California’s. It is not. It is $2,145/student above the average Unified School District revenue/student in California and is above the national average according to the DOE. These falsities are not acceptable and people should not say a thing unless they know what they are talking about and have the documentation to back up what they say.
Bruce Baker:
http://schoolfinance101.wordpress.com/2013/02/25/what-does-the-new-york-city-charter-school-study-from-credo-really-tell-us/
Ms. Ravitch, thanks to you, the CREDO national study is by far the most famous piece of charter research ever done. Everyone on the anti-choice side repeats your ubiquitous quote about how CREDO proves that only 17% of charter schools are better.
Do you distrust the CREDO scholars and their methods now?
“anti choice” side.
Love to hear from a spin master.
Too funny.
Diane had the open-mindedness and courage to publicly reframe her opinion on these issues, which, according to your snide bark at her heels, is a shortcoming of some kind.
As John Maynard Keynes once said, “When the facts change, I change my opinion. What do you do, sir?”
Three points:
1) The earlier nationwide CREDO study shows that AT MOST 17% of charters nationwide outperform non charter public schools. That study sets a ceiling for how well charters are doing. But it does not contradict the fact that even in those few cases where their methodology of invented students claims that a group of charters is doing better that claim may be false. In the case of New York City the data quoted in the post seems to indicate that the charters are not doing as well as claimed.
2) The data used in the post does look at student growth as defined by New York City and looks at the growth at the school level across 2 years which would involve the same students (except the ones charter schools have kicked out in the interim for not doing well on exams). The school to school comparisons used in the post, while an imperfect measuring device and used in seriously flawed ways in New York City to make decisions about school closures, are certainly less off than the “virtual student” methodology used by the Walton funded CREDO group as pointed out a long while back. See here http://nepc.colorado.edu/thinktank/review-multiple-choice
3) I too look forward to the Walton funded researchers response to Bruce Baker’s multiple posts showing that NYC charters are a shell game and don’t represent genuine educational reform and improvement. I may be in for a long wait…
I always liked Bruce Baker, but now he is a bit of hero for me after
his recommendations in his
A Not So Modest Proposal: My New Fully Research Based School!
http://nepc.colorado.edu/blog/tag/1464 (list of Bruce Baker blog posts with links)
There are a lot of good recommendations, but my favorite:
“Hire and keep only those teachers who have exactly 4 years of experience
“First, and foremost, since the research on teacher experience and degree levels often shows that student value-added test scores tend to level off when teachers reach about the 4th year of their experience, I see absolutely no need to have teachers on my staff with any more or less than 4 years experience, or with a salary of any greater than a 4th year teacher with a bachelors degree might earn.””
It would go well with Mark Naison’s
Why School Boards Love Temporary Teachers
http://www.laprogressive.com/school-boards-love-temporary-teachers/
“All over the country, school districts who do not have a teacher shortage — the most recent is Buffalo, New York — are trying to bring in Teach for America corps members to staff their schools. Why any school district would want to bring in teachers who have been trained for five weeks and have no classroom experience to replace teachers with years of training, experience, and mentoring would seem to defy common sense unless one considers the budgetary considerations at stake.
“Since few Teach for America teachers stay beyond their two-year commitment in the schools they are assigned to, there is a huge saving in pension costs for using them over teachers likely to stay till they are vested. Having a temporary teaching force also gives a school board greater flexibility in assigning teachers, and in closing old schools and re-opening new ones. It also, in the long run, will totally destroy the power of teachers unions in the district, allowing for costs savings that can be invested in increased testing and evaluation protocols.”
It might be useful to weigh in on the differences between the “DOE insider” claims and the results of the CREDO study of NYC charter schools. The critique that Diane has posted here looks at overall achievement of students in NYC charter schools versus “similar schools”. Achievement is defined as the measure of how much a student knows at a particular point in time, typically as measured on state tests. The insider analysis looks at school-level results of achievement between charter schools and similar schools.
The CREDO approach differs in two important ways. First, we look at how much a student’s academic knowledge grows from year to year. Growth is viewed by many researchers, including those engaged in this discussion, as a superior way of looking at a school’s impact on students. This is because absolute achievement is a combined result of prior learning, individual capacities, differences in background and the teaching that a student receives; clearly, not all of that should be attributed to a school when assessing their effectiveness. By focusing on growth, the influence of the remaining contributors to achievement are neutralized, yielding a clearer signal of school effectiveness.
Second, we see from this study and from earlier work that the student make-up in charter schools looks different in important ways even from the schools in their surrounding area. (There’s a table in the NYC report that lays out these contrasts.) Thus, school-to-school comparisons are not accurate — this is what led us to develop the VCR approach to compare student-to-student.
I wonder why no-one has raised the possibility that both the DOE insider and CREDO could be right.
Macke,
Thanks for commenting.
I hope you will also respond to Bruce Baker’s analysis of your report.
There are other issues that are outside the scope of your study that trouble many New Yorkers: the divisiveness and disruption caused by co-locations; the preferential treatment accorded to charters by public officials, which some call “academic apartheid”; the overloading of students with severe disabilities and low test scores in public schools because most charters don’t want them.
Diane
Well, I’m thoroughly confused. Macke, do you think the Times mischaracterized the CREDO study at all?
@Macke Raymond
You said
“This is because absolute achievement is a combined result of prior learning, individual capacities, differences in background and the teaching that a student receives; clearly, not all of that should be attributed to a school when assessing their effectiveness. By focusing on growth, the influence of the remaining contributors to achievement are neutralized, yielding a clearer signal of school effectiveness.”
Suppose during the year a kid gets cancer and he’s in hospital and recuperating at home for 6 months. Now that pretty much zero growth for that year (or maybe even a loss in learning). That zero growth is attributed to the school and to the teacher when that outcome has nothing at all to do with quality of the the teacher or the school.
So how does analysing growth fix that up?
So, you may say that it will all wash out because good things may happen to some kids that offset the bad things that happen to others. But bad events can be geographically centred, e.g. shootings in a school, floods in a state; and they happen with worse outcomes to poor families as they don’t have the resources to get themselves out of the predicament.
So how does analysing growth fix that up?
CREDO’s comparisons are as reliable as comparing the gas mileage of rented & leased cars. Not a meaningful comparison.
I’ve been in many NYC district schools. They vary enormously, from schools with no admissions tests, open to all, to schools that have very tough admissions tests. They (district schools) vary in terms of philosophy, curriculum, instructional approach, etc. etc. Same true for charters.
The amount of $ spent “comparing” district & charters is a waste of time and cash.
Far better to learn, for example from some of the exciting work on assessment that places like Central Park East, Julia Richman and other progressive district schools are doing. El Puente Academy for Peace and Justice is another terrific district school. http://elpuente.us/ I think there are charters in NY that also are worth learning from.
CREDO doesn’t get it. But some folks do recognize we can learn from each other, rather than spent time and money in questionable comparisons.
The bottom line for me in this is that looking at the comments from actual NYC residents and NYT readers below the Times article, they get it. They know not just how flawed the CREDO study is, but why it was published as reliable info by the Times in the 1st place. As to the CREDO study using virtual students, why would that have been done when the DOE had actual data that could have been used by CREDO? The link posted above to the Bruce Baker analysis is well worth the read.
Dear Citizens Arrest — the CREDO study is based on real actual data — of the exact same type used by DOE to get their academic measures. Their school grades include some other data that we do not use — or a more observational nature. But as far as judging the academics of schools, we use the same data in different ways. You can find out more about the study design that CREDO follows at credo.stanford.edu.
But isn’t the point that you match on quite wide variable ranges – like
“reduced or free lunches” when the charter kids in that category are more likely to be “reduced” than public schools kids. Similarly, with disability status – the kids in charter schools are more likely to have less severe disability than the kids in public schools.
AArgh and I like the NYT! I just wish they would do a better of interpreting data, not only in this case but others too. Maybe Ezra Klein can do a graph for us!!
If anything, it is obvious that charter schools aren’t performing miracles. Shut them down and allow the public schools to stay open instead of having the money transfered into a CEOs hands. It is time to put an end to this sham. I swear they put the same bs spin out about Michigan charters. Just stop please. I am still surprised that politicians have allowed taxpayer money to be completely ripped off. I think of the millions upon millions being wasted. We pay tax money and the politicians funnel it into private companies that are literally ripping off the public.
Incompetent and unqualified CEOs with no checks on their power make incompetent decisions. They hire unqualifed people who make incompetent decisions. The woeful leadership leads to massive teacher turnover. Could someone tell me why this model has been allowed to expand in the U.S.? Even with the ability to throw kids out at will, screen applicants, etc. they still stink. My description tells you why. I don’t have to read CREDO. I know chartes are not an answer to educational issues in the U.S. I think what bothers me most is that these schools have been promoted as some kind of miracle in tough urban environments. They are not. NY Times how desperate are you for money?
IGNORING THE ENCLAVE EFFECT
Statistical studies, esp. ‘quasi-experimental’ ones, are unreliable.
One of the biggest problems is that they do not address peer effects —
it is one of many, but it is constant. Think of your own child —
does s/he do better in a class of well-behaved, diligent students,
or does s/he excel in a classroom where a good percentage of the students
have trouble focusing and are a distraction?
This was the biggest problem with the C. Hoxby study of a few years ago that had a quasi-experimental design.
You may remember it — The New York Daily News touted its findings, saying, “It’s official. From this day forward, those who battle New York’s charter school movement stand conclusively on notice that they are fighting to block thousands of children from getting superior educations.” (Editorial, “Acing the test: Charter school students outperform peers by a mile in a fair test,” The New York Daily News, 23 September 2009)
Hardly.
What Hoxby’s study did and what quasi-experimental studies do most of the time is to compare students without addressing peer effects. Hoxby’s study compared winners of the lottery who go to charter schools with losers who don’t. This is random process, so the groups should be the same, more or less. The losers form a control group that has roughly the same socio-economic status, level of parental involvement, etc. That is all well and good as far as it goes, but it does not go very far because the peer effects are totally different. Just do a thought experiment: take the same kid, put him or her in with a bunch of kids who have a history of doing their homework and raising their hands, then put the same kid in a classroom with kids who have a history of not paying attention, throwing desks and maybe starting fires. (I taught in the Bronx and Roxbury – I do not exaggerate.)
The use of lotteries means that the kids who win are grouped together with other kids who win – in Diane Ravitch’s term, there is a siphoning effect.
More important, however, may be the ENCLAVE EFFECT. The kids who lose are grouped with a much more diverse student body that includes the kids of parents who did not know about or did not bother with the lottery. Sara Mosle made perhaps the most astute observation I’ve read of Waiting for Superman: the real heroes were the parents. If the influence of all those parents is concentrated in a few schools, of course they’ll do better.
I think the older CREDO study (the one that said 46% of charters were no better, 37% were significantly worse and only 17% were significantly better than public schools) was probably flawed as well, but its most important finding was ignored. That CREDO report offered a mixed picture of charters, with hugh variation by state.
It went on to say that the variation IN QUALITY “TENDS TO CORRELATE WITH the QUALITY OF CHARTER OVERSIGHT AND THE STRUCTURE OF CHARTER SCHOOL LAWS” (Thomas W. Carroll, “An Examination of What the CREDO Charter School Study Does and Doesn’t Show,” Foundation for Education Reform and Accountability (FERA), August 20, 2009; ANNOYING ALL CAPS ADDED.)
it is not that finding, but the national averages that get the most play in the press, even by noted and respected educators
As Jeff Henig has said, “advocacy groups appear anxious to enlist social science evidence and researchers to add legitimacy to their causes. . . . opposing cliques are ready and able to muster their own stable of researchers and findings to buttress their claims and challenge those cited by the other side.” In addition, “the public spectacle of research leads [many] to question whether the payoff is worth the hype, The seeming malleability of evidence reinforces cynicism about the independence and potential contribution of good scientific techniques.” (Jeff Henig, “The Spectrum of Education Research,” in ‘Data: Now What?’ Special issue of Educational Leadership, December 2008/January 2009, )
Final note — I try to limit mentioning my book, but if anyone is interested in an extended treatment of the above, it is all taken from
Respect for Teachers
or
The Rhetoric Gap and How Research on Schools is Laying the Ground for New Business Model in Education
On Amazon, or better yet, at
https://rowman.com/ISBN/9781475802078 (use the promo code RLEGEN12)
ALTERNATE UNIVERSE THOUGHT EXPERIMENT
One last thought: the problem with Charters is not the idea of a Charter school.
The problem with Standards are not the standards.
The problem with both is that opportunists seize on them in order to advance their agenda.
Do a thought experiment,
which seems to be at least as valid as a ‘quasi-experimental’ statistical study.
If, in some alternate universe, there was a group of wise public policy makers who had NOT been buffaloed into joining the stampede that said public schools were ‘failing’ even though 80% of them do fairly well and the other 20% suffer from immense problems that come from outside the school. Suppose, furthermore, they approached teachers and other educators as we treat firefighters — that is, they did not blame the teachers for the problems they faced any more than we blame firefighters when they have to battle a large fire. Instead, they would give teachers with the greatest exposure to social dysfunction the greater share of resources.
What do you think this group of wise women and men would do?
About Charters I think they would say this —
It is an interesting framework for innovation by experienced educators.
We want to avoid the charlatans coming in, so we will make sure the educators
working in these schools have the same salary scale, rights and protections from abusive administrators that the bulk of our teachers have.
We would want the administrators to be experienced and respected educators
who have a track record and the opportunity to open a Charter should be seen
as a reward and honor for a job well done over a long period of time.
Certainly we do not anyone using this framework for innovation as a way to cut corners. And certainly we do not want these Charter schools to expand too rapidly.
Rapid expansion would be reckless and harmful, resulting in poor quality and
hurting students in the long run.
Indeed, we should set a limit on how many charters are developed at the same time. Overall, we should limit them to 5 to 10% of the student body for now, see how they do, see what effects they have on the public schools (for we don’t want this to be a zero-sum game in which the Charters improve at the expense of the public schools) and then proceed from there.
As for Standards, I think they would say —
Standards can be helpful, but they can be stifling. There is always someone who wants to add something else to the list of things to be learned and then a teacher –the professional to whom we entrust our children’s growth– may feel obliged to cover everything. Covering a topic is not educating, so we must allow our teachers some level of autonomy, not have them overburdened by the task and feel the pressure of administrators. As Aristotle said, leisure is the condition necessary for the pursuit of philosophy, by which he meant the love of wisdom. We must trust our teachers to follow what they believe is the best path.
And, most importantly, we cannot expect that we will be able to devise an adequate measurement system for what we want to happen in the child’s life. As Jean Piaget told us, “Anyone can confirm how little the grading that results from examinations corresponds to the final useful work of people in life.” The last thing we need it for someone to come along and standardize the process by monitoring all the time. The effects of monitoring are not neutral — they limit creativity and push us to the more quantifiable when it is quality that we seek.
Above all, we cannot put pressure on our students in the forms of high-stakes tests.
To do so will corrupt the standards, which need to be internalized by educators and then by students, not enforced by an outside authority.
Please note: In this alternate universe Arne Duncan is coaching basketball, successfully, I hear. People say it was what he was always meant to do.
Diane,
The NYTimes article mentions “The city is also a magnet for education talent, drawing successful charter school management organizations, like Kipp and Uncommon Schools, that can replicate good instructional techniques.” I noticed that those two “super-networks” performed better, at scale, than TPS in the Credo study. What are your thoughts on the large-scale success of those “super networks,” is there something we can learn from them?
Thanks,
John
Yes. As Bruce Baker and Julian Heilig Vasquez have shown, these charters spend more than district public schools. That is an important lesson.
Charter schools equation for success does not work. What more is there to say so you must ask the question “Why are they supporting them?”