Mark Weber, aka the blogger Jersey Jazzman, is getting his doctorate in research and statistics while teaching in a public school in New Jersey. He is a sharp critic of shoddy research, especially when it comes to the fantastical claims made on behalf of charter schools.
In his latest post, he asks why CREDO, the charter-evaluating institute at Stanford University run by Macke Raymond, continues to use a metric that has never been validated.
Journalists who have little expertise in evaluating research claims eagerly take up the claim that School X produces an additional “number of days of learning.”
It happened most recently in Texas, where charter schools finally managed to match the test scores of public schools (you know, those “failing schools” for which charter schools are supposed to be the rescuers.)
He shows how the Texas study refers to “days of learning” and this is translated to infer “substantial” improvement. But, as JJ shows, the gains are actually very small, and might more accurately be described as “tiny.”
He writes:
Stanley Pogrow published a paper earlier this year that didn’t get much attention, and that’s too bad. Because he quite rightly points out that it’s much more credible to describe results like the ones reported here as “small” than as substantial. 0.03 standard deviations is tiny: plug it in here and you’ll see it translates into moving from the 50th to the 51st percentile (the most generous possible interpretation when converting to percentiles).
I have been working on something more formal than a blog post to delve into this issue. I’ve decided to publish an excerpt now because, frankly, I am tired of seeing “days of learning” conversions reported in the press and in research — both peer-reviewed and not — as if there was no debate about their validity.
The fact is that many people who know what they are talking about have a problem with how CREDO and others use “days of learning,” and it’s well past time that the researchers who make this conversion justify it.
Jersey Jazzman calls on Macke Raymond and the staff at CREDO to justify their use of this measurement. The “days of learning” inflates the actual changes, he says.
The concept of days of learning, he says, is based on the work of economist Erik Hanushek of the Hoover Institution (Stanford). It may be coincidental that he is Macke Raymond’s husband. They are both very smart people. I hope they respond to Mark Weber’s challenge.
Diane Ravitch's blog 
Where can we find a CREDI?
LikeLike
It’d be a hell of a lot more CREDIble than CREDO.
LikeLike
Now that privateers have driven a wedge into gaining access to public funds, we must be vigilant about questioning the assertions and “alternate results” of studies conducted by charters or their sympathizers. We have to be ready to question and approach their findings with a healthy dose of skepticism.
LikeLike
Continued use of a metric that has never been validated quacks like the use of a
metric (test scores) that has been invalidated (WILSON).
If the oppositional energy remains caged in metrics, slogans, and myths, the
“Lead roles in a cage” will continue.
LikeLike
Notice the names these groups use to ‘name” themselves … and to disguise their purposes. SPIN, SPIN, SPIN.
LikeLike
I posted this comment on Mr. Jazzman’s blog. Diane already clarified the Hoover connection here and the Raymond-Hanushek relationship.
Hi Jersey Jazzman. Aside from the content here, it must be pointed out that CREDO is not part of Stanford University itself, not a scholarly research program as that implies. CREDO is a program of the Hoover Institution, a right-wing, free-market “think tank” (aka propaganda operation) that is located on the Stanford campus. CREDO formerly described itself as a program designed to PROMOTE charter schools, and used to be open about being run by Hoover (NOT Stanford). The materials about Hoover and CREDO have become more and more cagey about the connection, so now it’s almost impossible to discern that. Also, CREDO’s director, Hoover fellow Macke Raymond, is married to Eric Hanushek, whose “research” she cites. Hanushek is also a Hoover fellow whose longtime specialty is promoting propaganda aimed at disparaging and discrediting teachers.
So, just be aware that describing CREDO as part of Stanford and implying that it’s a scholarly research project rather than a propaganda operation helps promote its credibility.
That said, I do recognize that CREDO has produced some research that didn’t reflect favorably on charters, so it seems to zigzag. But still, it needs to be characterized clearly and accurately.
The very term “think tank” is in itself a piece of propaganda, as these operations are propaganda operations, not scholarly research organizations.
Disclosure that I did a large freelance writing job for the Hoover Institution many years ago, so I gained quite a bit of familiarity.
LikeLike
Thank you – this is excellent information.
I don’t think CREDO has produced anything unfavorable to charters in quite a few years.
LikeLike
I REALLY appreciate folks who dig into the research and question the assumptions. I always wondered where that “days of learning” metric came from . Once again, relying on standardized testing as a gauge of learning has been shown to be unreliable.
LikeLike
“relying on standardized testing as a gauge of learning has been shown to be unreliable.”
Standardized testing has been shown to be completely invalid by Noel Wilson. Without validity, nothing else matters. A test can “reliably” be not able to assess what it purports to assess. That is just one of the many indicators of the invalidities involved in the standards and testing regime. Questions of validity are the most important as shown by Noel Wilson. For a short discussion on that invalidity see: http://edrev.asu.edu/index.php/ER/article/view/1372/43
LikeLike
“days of learning”
Early in the 1980s, soon after I started teaching, I was teaching a 7th grade English class in a middle school with a child poverty rate that must have been close to 100 percent. The principal warned us, teachers, to never leave the campus on foot due to the gang violence in the community around the school. His warning was, if we left the campus on foot, we might vanish and never be seen again.
Anyway, I had one mother ask me for advice because her daughter was testing five years behind her grade level in literacy. She asked me what could be done to catch her daughter up.
I told that mother to turn off the TV and have a reading time for the entire family. Read books every night for at least an hour and then gather and talk about what they were reading. That was early in the school year. She came to see me after the next round of tests that also reported literacy levels, and she told me her daughter closed the gap in one year by turning off the TV, visiting the library where the books were free, and having a family reading hour (or longer) every night seven days a week.
That is the real meaning behind “days of learning”.
The sad thing is that in 30-years of teaching thousands of students, she was the only parent to ask me for advice who then did what I suggested. Such a simple solution to turning every day into a learning day and have fun doing it.
LikeLike
Mark Weber is more qualified than I am to criticize the statistical leaps through thin air that CREDO routinely uses to portray ”days of learning” gained or lost by students who attend this or that kind of school.
The latest report from CREDO is titled “Charter Management Organizations:
2017,” funded by the Walton Family Foundation and Fischer Fund. It is not peer reviewed, but it is available online. https://credo.stanford.edu/pdfs/CMO%20FINAL.pdf
This report has 68 references to the fictional “days of learning” metric. In this and in many other reports, CREDO researchers show themselves to be really lazy data-munchers. Here, “lazy,” means that the researchers only track math and reading scores and changes in these, year to year. They are perfectly self-satisfied in defining “Growth” as: “The year-to-year change in academic performance relative to one’s peers. Growth can be positive or negative.” They are not at all troubled by stripping down the meaning of “academic performance” to the ups and downs of scores in two subjects.
In this report the authors introduce some tweaks in the way those fictional “days of learning metric” are conjured (p. 12). They offer this explanation of their use of the phrase: “We wish to emphasize that the days of learning translation is only meant to be a loose approximation of the effect size to provide a sense of scale to aid the reader in interpreting the effect sizes. The effect sizes are the mathematically computed measures produced by the statistical models and should be the basis for policy decisions (p. 12)”
I think readers should ignore the arrogance in this appeal to the authority of statistical models and mathematically computed measures as the basis for policy decisions. In this case, CREDO researchers seem to think that readers of their research will accept the phrase “days of learning’ as good enough—simple, easy–more accessible than thinking about effect sizes expressed as standard deviations in test scores.
Do not be fooled by this condescension. Anyone who has some reality-based knowledge of learning and education will see this as a misleading phrase. It is as misleading for the charter industry and for public schools. Lloyd gives an excellent example, but here is a different take.
First, there is no national requirement for the number of days that a student must study, and only study reading or only study math. Math and reading are not the only subjects studied in a typical school day. Moreover, how much time a student spends studying math or reading—whether expressed in minutes, hours, days, or years—is not the same as how much students have actually learned, or whether that “much” is valued, or whether learning is valued for compelling reasons. Full value added is not represented in test scores.
Second, there is no national requirement for the number of instructional days in a school year. The national average for public schools is 180 days, but Alaska requires 170; Michigan, 165; Colorado, 160. In some states an instructional day is counted even if students are not present but staff time is devoted to professional development or conferences with parents. Instructional time is devoured by too many days reserved for test prep, proctoring tests and make-up tests, and special arrangements for testing students who are learning English or in special education programs.
Third, the length of time in an instructional day typically varies by grade level, and that length may include or exclude lunch and recess/break time. There are two main patterns in these specifications. One is a school day defined as six hours for all grades. The District of Columbia includes lunch and recess, while New Jersey excludes lunch and recess. Both claim the day is six hours long. In New Jersey, the minimum time for an instructional day is actually truncated to four hours. A more typical pattern is for the length of the day to differ by grade span but with different grades included in a span. For students in grade three, the instructional day may be as little as 4.5 hours in Ohio but 6.5 hours in Tennessee.
So far, all of my citations have come from a database maintained by the Education Commission of the States. The curators include in the database the specific state statutes on time allocations when these are available. http://www.ecs.org/clearinghouse/95/05/9505.pdf
Fourth, CREDO’s references to “days of learning” fails to consider that real students and their teachers have excused and unexcused absences from school. The researchers also fail to recognize that states, districts, and schools may have expanded learning time programs, and these programs may or may not be available or recommended for every student. CREDO’s 2017 report on Charter Management Organizations does not look at absentee rates in the schools it studied, in the “virtual twins” it conjured, or online charter schools–where student enrollments and days of learning are deliberate fictions, as in Ohio.
The “days of learning” reported in CREDO research has absolutely no relationship to real-world teaching and learning. Period. End of story.
LikeLike
In addition, “days of learning” should also take into account the work a child does to learn during the day.
Children who warm a seat but don’t do the classwork, don’t do the homework or read outside of school are not going to show as much growth as a child that does all the classwork and homework and reads outside of school as an avid reader of books like Harry Potter.
The children that don’t do the work and don’t read are the first ones that most if not all Corporate Charter schools are going to get rid of and that has an impact on that flawed comparison.
Because the fake reformers never mention the fact that the most difficult children to teach, the ones who do not cooperate in class or do homework or read on their own, end up mostly in traditional public schools, the assumption is that all children are making the same effort to learn and that the corporate charter schools are doing a better job with those children.
And another Stanford study from 2013, outside of CREDO points this out – that this challenge exists in every country even Finland but more so in the United States because the U.S. has more children living in poverty where most of these difficult to teach children are found.
“Poor ranking on international test misleading about U.S. student performance, Stanford researcher finds”
The report also found:
U.S. PISA scores are depressed partly because of a sampling flaw resulting in a disproportionate number of students from high-poverty schools among the test-takers. About 40 percent of the PISA sample in the United States was drawn from schools where half or more of the students are eligible for the free lunch program, though only 32 percent of students nationwide attend such schools.
http://news.stanford.edu/news/2013/january/test-scores-ranking-011513.html
LikeLike
Nepotism is the new rising strategy for success.
Who better than ones family to give unconditional support when all your actions are on the wrong side of an issue?
LikeLike