The National Education Policy Center regularly reviews research findings, in effect, acting as an independent peer review board.
In this case, its reviewer challenges the latest CREDO report on urban charters:
Is It Time to Stop the CREDO-Worship?
New review explains CREDO charter school research flaws, raises concerns about misunderstandings of effect sizes
Contact:
William J. Mathis, (802) 383-0058, wmathis@sover.net
Andrew Maul, (805) 893-7770, amaul@education.ucsb.edu
URL for this press release: http://tinyurl.com/mbse6m7
BOULDER, CO (April 27, 2015) — A recent report contends charter schools generally helped students increase reading and math scores and that urban charters had an even stronger positive effect. But a new review released today questions the strong reliance that has been placed on this and similar reports.
Andrew Maul reviewed Urban Charter School Study Report on 41 Regions 2015 for the Think Twice think tank review project. The review is published by the National Education Policy Center, housed at the University of Colorado Boulder School of Education. Maul, an assistant professor in the Graduate School of Education at the University of California-Santa Barbara, focuses his research on measurement theory, validity, and research design.
Urban Charter School Study Report on 41 Regions 2015 was produced and published by the Center for Research on Education Outcomes (CREDO) at Stanford University’s Hoover Institution. It is a follow-up report to CREDO’s 2013 National Charter School Study.
The new report analyzes the differences in student performance at charter schools and traditional public schools in 41 urban areas in 22 states. Researchers sought to establish whether being in an urban charter school, as opposed to a non-urban one, had a different effect on reading and math scores, and if so, why. The report found a small positive effect of being in a charter school overall on both math and reading scores, and a slightly stronger effect in urban environments.
Maul’s review, however, explains “significant reasons to exercise caution.”
For its analysis, CREDO again used its own, unusual research technique that attempts to simulate a controlled experiment: constructing “virtual twins” for each charter student. The “twins” were derived by averaging the performance of up to seven other students, chosen to match the charter students by demographics, poverty and special education status, grade level, and a prior year’s standardized test score.
Maul points out that the technique isn’t adequately documented. He adds: “It remains unclear and puzzling why the researchers use this approach rather than the more accepted approach of propensity score matching.” The CREDO technique, he warns, might not adequately control for differences between families who select a charter school and those who do not.
CREDO also fails to justify choices such as the estimation of growth and the use of “days of learning” as a metric.
But regardless of concerns over methodology, Maul points out, “the actual effect sizes reported are very small, explaining well under a tenth of one percent of the variance in test scores.” The effect size reported, for example, may simply reflect the researchers’ exclusion of some lower-scoring students from their analysis.
“To call such an effect ‘substantial’ strains credulity,” Maul concludes. Overall, the report fails to provide compelling evidence that charter schools are more effective than traditional public schools, whether or not they are located in urban districts.
Find Andrew Maul’s review
on the NEPC website at:
http://nepc.colorado.edu/
thinktank/review-urban-
charter-school
Find CREDO’s Urban Charter School Study Report on 41 Regions 2015 on the web at:
http://urbancharters.
stanford.edu/index.php
The Think Twice think tank
review project (http://thinktankreview.org)
of the National Education Policy Center (NEPC) provides the public, policymakers, and the press with timely, academically sound reviews of selected publications. NEPC is housed at the University of Colorado Boulder School of Education. The Think Twice think tank review project is made possible in part by support provided by the Great Lakes Center for Education Research and Practice.
The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence.
For more information on the NEPC, please visit http://nepc.colorado.edu/.
This review is also found on the GLC website at http://www.greatlakes
center.org/.
The National Education Policy Center (NEPC) is housed at the University of Colorado Boulder School of Education. Its mission is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information about the NEPC, please visit http://nepc.colorado.edu/.
Our mailing address is:
National Education Policy Center
School of Education, 249 UCB
University of Colorado
Boulder, CO 80309-0249
For all other communication with NEPC, write to nepc@colorado.edu.
National Education Policy Center · School of Education, 249 UCB · University of Colorado · Boulder, CO 80309-0249 · USA
Diane Ravitch's blog 
Reports such as the CREDO study emphasizing the relative advantages of charter schools or the study by NEPC questioning the advantage claims are important. However, they beg the question of whether charter schools are sound education policy if scalable systemic equity is the goal. Charter schools fail as a policy because even their relative success depends on competition, the foundation of which is individual choice not widespread improvement.
Once advocates for democratic equitable education engage in the debate from the framing of which choice of schools is best for an individual child, we lose. There will always be better or worse individual choices. We need education policy that prioritizes improvement for all students as the most reliable route to improvement for individual students.
http://www.arthurcamins.com
This is the exact same methodology used in the 2009 CREDO study that Dr. Ravitch used to talk about at every opportunity, as if it were the only charter school study in existence.
Funny how Dr. Ravitch suddenly became interested in methodological critiques as soon as CREDO found charter school advantages.
WT,
As a general rule, it’s pretty useful to ignore all studies, regardless of their findings, regarding charter and public school comparisons. Just look at the funders and researchers and you’ll know which way it will go.
To be honest, I don’t think a fair comparative study can be made between the two. Each type of school is playing a different game, so to speak. As long as both sectors don’t have to play by the same rules, what’s to compare?
It’s like comparing baseball and cricket. There are similarities. A field, a ball, a bat, a pitcher. But it doesn’t translate that a good cricket player will have the same success at baseball. They aren’t identical.
For example, the CREDO study attempts to do virtual twins but does not take peer effects into account. Charter schools are full of children from functioning families. Traditional publics have some children from functioning families. (Many charter parents that I’ve spoken to have noted that the make-up of the student body was a primary driver in their choice.)
So yes, you can have fun discussing Ravitch’s criticisms of CREDO reports, but the truth is that educational research is rarely objective.
Unlike Arthur Camins above, I think that you miss the bigger point.
Let’s start the conversation here.
A,C……”There will always be better or worse individual choices. We need education policy that prioritizes improvement for all students as the most reliable route to improvement for individual students.”
Parents have always had the right to choose private and/or religious education for their own children. The discussion regarding how our public monies are spent on education is the discussion that we need to be having. How do we address the question of equity for all students?
Charter schools and vouchers can never be the answer, or even part of the answer to a long term solution for providing equitable funding for the education of ALL children.
Charters fail in the goal of “scalable systemic equity.” Unfortunately, the same is true for many public schools. Schools need more opportunities for diverse students to learn together in a school that is clean, safe, and equitably funded and resourced. States need to be held accountable for funding disparities among districts.
So are you trying to discredit Dr. Ravitch, the 2009 report, this blog, or just having a bad day?
Math Vale: I am surprised and amazed—no, stunned and horrified!—that you have incorrectly written this upcoming item on a Pearson standardized test because:
You have not included one “distractor” or “mislead” or “decoy.”
All the answers are, psychometrically speaking, “best answers.”*
*Rheeally! And in the most Johnsonally sort of ways to boot.*
¿😳?
Unless, of course, you add a fifth response: “E. All of the above.”
My bad. To paraphrase that renowned Mexican superhero of yesteryear, El Chapulín Colorado [The Red Grasshopper]—
“No contaba yo con su astucia.” [I didn’t count on your astuteness.]
I promise to pay more attention next time.
😎
WT is exactly right.
Here’s what NEPC said about CREDO (same methodology) in 2009: “The relative strength and comprehensiveness of the data set used for this study, as well as the solid analytic approaches of the researchers, makes this report a useful contribution to the charter school research base”
Terrific find. I also vaguely recalled seeing some happy Tweets from NEPC when the Ohio CREDO study was released, and sure enough: http://nepc.colorado.edu/blog/bad-news-ohio-charters
Tim,
Yes, that one even more significant because they had no issue reporting it when students had fewer “days of learning”.
Pretty much all of the recent studies are consistent: CREDO, Hoxby, Mathematica. I doubt we’ll ever see acknowledgement of that on this blog, but I think that saying charters underperform is just not borne out by the facts. It’s probably too early to say that they overperform, but I think the data is there that they do in urban areas and with minority students.
WT, John, Tim, Steve, Betsy, RT, MV, KTA, etc. . ., in other words all of the readers here and everywhere:
Judging schools, districts and/or countries on the basis of standardized test scores amounts to a bunch of mental masturbation and has absolutely nothing to do with the “quality” of public, charter, private or whatever kind of schools and the instruction (the teaching and learning process) that the students experience. The data coming from those vaunted standardized tests is/are so corrupt and conceptually error ridden that any conclusions are COMPLETELY INVALID, or, as Noel Wilson puts it “VAIN AND ILLUSORY”.
Why we put so much time, effort and resources to gathering ABSOLUTELY INVALID INFORMATION is beyond me. The sooner we quit abusing and experimenting on the most innocent of society, the children, the sooner we will get to providing better, more equitable teaching and learning processes for all children.
To understand these very simple ontological and epistemological problems with educational standards and standardized testing see below (so as to not Chiletize the thread) for the link and my summary to Noel Wilson’s never refuted nor rebutted seminal treatise on these issues.
Of course Duane, but not everyone can hear what you are saying in quite the same way.
And some who come here do not want to listen and converse, they merely want to win the argument.
…. or divert the conversation.
Reblogged this on Crazy Normal – the Classroom Exposé and commented:
“To call such an effect ‘substantial’ strains credulity,” Maul concludes. Overall, the report fails to provide compelling evidence that charter schools are more effective than traditional public schools, whether or not they are located in urban districts.
Continuing from above:
“Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine.
1. A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other word all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.”
The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
Thank you Betsy and Señor Swacker:
To people who come here with an intention which is not to listen and converse, but merely to win the argument or to divert the conversation.
Please re-read carefully WHAT Dr. Ravitch supports and REMINDS all readers in this website that “UNITED WE STAND, DIVIDED WE FALL”
1) We support public education because it is a pillar of our democratic society.
2) We support schools that are subject to democratic control by members of their community.
3) We support the principle that every classroom should be led by a teacher who is well educated, well prepared for the challenges of teaching, and certified.
4) We support teacher professionalism in decisions about curriculum, teaching methods, and selection of teaching materials.
5) We support schools that offer a full and rich curriculum for all children, including the arts, physical education, history, civics, foreign languages, literature, mathematics, and the sciences.
6) We support parent involvement in decisions about their children.
7) We support the idea that students’ confidential information must remain confidential and not be handed over to entrepreneurs and marketing agents.
8) We support schools that have the resources that their students need, such as guidance counselors, social workers, librarians, and psychologists.
9) We support schools that have reasonable class sizes, so that teachers have the time to help the children in their care.
10) We support high standards of professionalism for teachers, principals, and superintendents.
11) We support assessments that are used to support children and teachers, not to punish or stigmatize them or to hand out monetary rewards.
12) We support assessments that measure what was taught, through projects and activities in which students can demonstrate what they have learned.
We support the evaluation of teachers by professionals, not by unreliable test scores.
We support helping schools that are struggling, not closing them.
We support early childhood education, because we know that the achievement gap begins before the first day of school.
We support the equitable funding of schools, with extra resources for those students with the greatest needs.
We support wraparound services for children, such as health clinics and after-school programs.
Regardless of people who have intention and agenda of TWISTING or COMPARING in what Teacher Duane Swacker expresses as:
“Why we put so much time, effort and resources to gathering ABSOLUTELY INVALID INFORMATION is beyond me”
Please, It is time to remind the public that people with greed will not tolerate humanity, civility, and democracy. As a result, whoever works for people with greed, cannot be trusted.
Definition of greed = grab or loot public fund/assets with bribes.
Definition of virtue = patience, kindness, consideration, calmness, and genius (being born with, or years of working + learning experiences with conscience.)
In short, every time people who have twisted intention, please ask yourself a question,
Do we have a virtue to STAY AWAY any manipulative TRAP from GREEDY corporate backers or from people with knowledge, power, wealth but lack of CONSCIENCE, and TOLERANCE for humanity, civility, and democracy? Back2basic