In a recent post on the Shanker Blog, researcher Matt Di Carlo chastises those “reformers” who expect transformational results from educational interventions.
“A recent report from the U.S. Department of Education presented a summary of three recent studies of the differences in the effectiveness of teaching provided advantaged and disadvantaged students (with the former defined in terms of value-added scores, and the latter in terms of subsidized lunch eligibility). The brief characterizes the results of these reports in an accessible manner – that the difference in estimated teaching effectiveness between advantaged and disadvantaged students varied quite widely between districts, but overall is about four percent of the achievement gap in reading and 2-3 percent in math.
“Some observers were not impressed. They wondered why so-called reformers are alienating teachers and hurting students in order to address a mere 2-4 percent improvement in the achievement gap.
“Just to be clear, the 2-4 percent figures describe the gap (and remember that it varies). Whether it can be narrowed or closed – e.g., by improving working conditions or offering incentives or some other means – is a separate issue. Nevertheless, let’s put aside all the substantive aspects surrounding these studies, and the issue of the distribution of teacher quality, and discuss this 2-4 percent thing, as it illustrates what I believe is the among the most important tensions underlying education policy today: Our collective failure to have a reasonable debate about expectations and the power of education policy.”
“Reformers” often boast about miraculous results but those results usually turn out to be the result of skimming, creaming, intensive test prep, or other kinds of manipulation of data. Real change is slow and incremental. Everyone know that but “reformers.”
Diane Ravitch's blog 
Meanwhile studies show that reducing class size in the early grades narrows the gap by about 30% — ten times as much.
Leonie Haimson
LikeLike
Let’s support our struggling schools with experienced teachers, positive role models, wrap-around services, and good leadership…We can do this for our children..
LikeLike
The only real transformations sought by the so-called reformers are to reconfigure labor relations, turning teaching into temporary, non-union, at-will employment; to privatize as much of it as possible; to control the curriculum and instruction so that it both enriches them and turns the schools into pipelines for their perceived labor market needs.
Everything else is cant and deception.
LikeLike
I worry that the small percentage of change attributed to teacher effectiveness could be used maladroitly to make the argument that “teacher cometency doesn’t matter (NC-style),” thus justifying TFA and other unskilled temp workers in classrooms instead of professionals.
LikeLike
“Competency.”
LikeLike
I’d never say teacher competency doesn’t matter, Alan, just not to the degree the “let’s put a great (meaning new, low-salary) teacher in every class!” reform bots keep repeating.
LikeLike
I thoroughly agree with the need to look for a true sense of (a) the realistic possible gain from deform along with (b) the realistic cost for deform.
I can’t speak about closing the gaps. But my own envelope estimate of potential educational gains are also in the tiny percentages…
…I assume that about half of what I need my kids to gain from school is what we might call Academic. The rest are good work habits, the ability to search for answer, the ability to work with others, …
…of the Academic half, about half of that is reading, writing or ‘rithmetic. (About 25% of the total).
…of the three R’s, only about 1/2 can be evaluated well with standardized tests. Just consider writing. The goal is for people to communicate well in writing. But the tests just gauge mechanics – not communication.
SO… Standardized tests apply to 12.5% of education.
What gain can we get? A maintainable 20% improvement on average test scores over a 10 year period would be absolutely amazing results.
So… 20% of 12.5%… That’s a MAXIMUM educational gain of 2.5%… And that assumes that testing programs don’t hurt anywhere else… They will. Most likely there will be a net harm to education. Or a gain of no more than 1%.
It’s reality of the limitations of testing.
A very bad equation for education. No smart business would invest billions for the high risk associated here. (Dumb businesses do dumb investments…but I don’t really think we want to make education run like a dumb business.)
LikeLike
Question:
Can a majority here agree on one term to describe the robber barons and wolves of Sesame Street?
How about “fake reformers”? I’ll even stop using robber barons and wolves of Sesame Street. Right now, there’s too many terms describing these fakes.
I think it’s an insult to the real “reformers” of the Teddy Roosevelt (TR) era to use anything but “fake reformers” whenever we refer to the robber barons and wovles. (I’m still reading “The Bully Pulpit” by Doris Kearns Goodwin.)
There is no way these billionaire oligarchs and their paid toddies deserve to be thought of as “reformers” as if they inherited the mantel from TR.
Or maybe “counterfeit reformers” or “mock reformers” or “phony reformers” or “fabricated reformers” or “bogus reformers” or “fraudulent reformers”)—-something simple, easy and difficult to forget once it becomes repeated enough to become a meme.
Let’s brand them with one term and then maybe the majority will stick to the brand when referring to them.
I looked for a one word antonym for reformer but couldn’t find one even through Google.
Just one term used as often as possible until it sinks into the public thought process and when a person sees the name Bill Gates (or one of the other wolves) they think of that meme. Just a suggestion.
LikeLike
It’s an insult to Sesame Street, too.
LikeLike
Depends on how you read it
One way it sounds like they are from Sesame Street (glass half empty view)
Another way it sounds like they are robbing Sesame Street (glass half full)
LikeLike
Welcome to the world of “insane reading growth” and edtech solutions:
https://www.edsurge.com/n/2014-03-18-how-one-teacher-achieved-insane-reading-growth-last-year
There’s a lot to make fun of here, although I am in favor of some of the methods mentioned in the article, especially the idea of believing in your students. It can’t hurt.
EdSurge receives funding from the usual suspects. If you want the latest edtech hype and venture capital news, you can subscribe to their newsletter.
LikeLike
Randal, as the article’s author, I wouldn’t call Tracy Fischetti’s success an “edtech solution”. It’s a pedagogical solution (differentiation and hitting the proximal zone of development have been around a long time) that uses some edtech to get there. Would love to hear what else you’d like to make fun of there, and see if I can offer any insights.
LikeLike
No, I’d rather just make fun of the title. It’s obvious the teacher is working hard to help her students improve. I’m excited about the promise of educational technology, just not the applications I usually read about on EdSurge.
LikeLike
On second thought, if you care about these kids, use some of your grant money to buy them some actual books! And ask your big donors to do the same thing. Book ownership makes a difference. And books cost lest than computer systems.
LikeLike
I’m interested in discovering what accounts for Fischetti’s success, in hopes of sharing that with other educators. A few academics have pointed to flaws in the theories behind the metacognitive techniques she uses, which may suggest that even imperfect metacognition can be better than none. That’s helpful.
If the title is the only thing you take issue with, well, I guess I can live with that. If you can help me refine my understanding of what is, and is not, the underlying cause of her success here, I’d be very interested.
LikeLike
I’m not really in a position to judge, so don’t quote me. I’ll just give my overall reaction. The “gains” could be the result of a committed teacher plus the Hawthorne effect. I’m skeptical that the kind of computerized system you describe will be of any deep and long-lasting value. In any case I doubt the sort of testing being done is completely legit, so I don’t put much stock in massive gains in a short time. There’s no telling whether the effect will last, or what the effect actually means. If improvement on tests is a result of familiarity with the equipment and format, then the effect may be meaningless. In any case, I don’t consider it authentic learning, which I would define as learning that a student is truly involved with, rather than subjected to. If, within the program, students are able to discover and choose materials that truly appeal to them, I might be more a little less skeptical. But too much time spent with a program like this will squeeze out other activities that might be more valuable (even though they may not raise scores.)
I’m unimpressed with the whole idea of lexiles and “the proximal zone of development.” My first instinct is to scoff. I became a good reader by being read to, having fun with Dick and Jane in first grade, having books recommended to me by my older brother, making visits to my small town’s storefront library, browsing the encyclopedia, and reading My Weekly Reader and Junior Scholastic from cover to cover. Other than some SRA reading cards in eighth grade, I had no programmed instruction that I can remember. That’s why I say kids should be given books. (See the work of Stephen Krashen and Dolly Parton–yes, that Dolly Parton.) I’m into libraries, not lexiles.
I’m all in favor of teaching kids to use reading strategies and assigning metacognitive activities, as long as this isn’t overdone. Some of these practices emerged from research done a few decades ago, much of it by the Centers for the Study of Reading. And I used some of them successfully when I taught. For example, helping kids engage their prior knowledge as they face new reading material isn’t a bad idea. It just depends how you do it. As a general rule, the younger the child, the more context needed. I think the backlash against these practices is overblown, especially by people like David Coleman, who misrepresents them in his description of the “five shifts” of the Common Core. Teacher discretion is the key to a balance between teacher “scaffolding” and student independence. My personal bias is in favor of independence, but it all depends on the students, the purpose of the lesson, and the lesson itself. (I’m actually not all that attached to the concept of “lesson,” though.)
Anyway, I’m all in favor of bringing in brain research, but hasn’t the right-brained/left-brained labeling been debunked to some extent? I have to admit I never heard of “mid-brained.” I labeled myself whole-brained until I heard that the whole labeling thing is an oversimplification, if not a complete misconception. To me, the most exciting discovery from brain research is the principle of neuroplasticity, which should make everyone believe they can get better at doing practically anything.
Bottom line: Congrats to the teacher, who is using lots of different modalities and trying new things in her classroom. It could be her personality that is the main reason for any gains in reading. I’m just very skeptical of claims like this. I’d be more impressed if a teacher reported that her students can’t get enough of school. The more programmatic the teaching becomes, the less that’s likely to happen.
LikeLike
Also, I don’t think you should publish articles like that one. If you have no idea what’s responsible for the teacher’s success, but you feature an educational technology product and endorse it to an extant, you’re not really doing anyone any favors. You’re just giving unearned credit to the maker of the product. If you’re serious about finding out what’s going on, get some of your funders to sponsor controlled studies by qualified researchers. (And then you’ll get some reputable people to review that research.) Anecdotes from suspicious people like me aren’t going to clarify anything. Even though the teacher may be doing a great job, the whole thing smells of hype.
LikeLike
From my thirty years in the classroom (1975 – 2005), teacher success comes more from bull headed, stubborn dedication than anything else.
What I mean is: Teachers who refused to cave in to pressure from politicians, administrators and parents obsessed with a child’s self-esteem; instead, teachers who do what’s best for the child’s future as an adult.
Unfortunately, not all teachers have a thick enough skin or are stubborn enough to stand against the massive weight of the pressure that comes at them almost everyday.
And if tenure vanishes, the number of teachers leaving and/or caving in to the politically correct pressure from any source will skyrocket. I suspect that the most dedicated teachers will leave for jobs with less stress.
LikeLike
Randall, thanks for the thoughtful response. A few answers:
“I’m skeptical that the kind of computerized system you describe will be of any deep and long-lasting value.”
First, Fischetti’s class is not that ‘computerized’; she simply uses some digital tech as a part of a broader strategy. Do you doubt the long-lasting value of differentiation? What if tech helps a teacher do that better?
—
“In any case I doubt the sort of testing being done is completely legit, so I don’t put much stock in massive gains in a short time.”
I agree what’s being tested is not what’s most important. Character and wisdom trump reading comprehension. But I do think there is some value in what can be measured. There’s a big difference between “important” and “most important”.
—
“If improvement on tests is a result of familiarity with the equipment and format, then the effect may be meaningless.”
I doubt that accounts for the entirety of the effect, but we should find out how much is attributable to familiarity, agreed. Fischetti doesn’t think it’s the main factor, and I think her opinion is a good place to start, so I’m not to worried about her results being meaningless.
—
“Too much time spent with a program like this will squeeze out other activities that might be more valuable (even though they may not raise scores.)”
It might also squeeze out other meaningless activities that would be less valuable. Each teacher will need to make that judgement. Giving them knowledge about what other effective teachers are doing is a good thing.
—
“I became a good reader by being read to, having fun with Dick and Jane in first grade, having books recommended to me by my older brother, making visits to my small town’s storefront library, browsing the encyclopedia, and reading My Weekly Reader and Junior Scholastic from cover to cover. Other than some SRA reading cards in eighth grade, I had no programmed instruction that I can remember. That’s why I say kids should be given books.”
Giving books to students who are already demoralized about reading because they’ve always struggled won’t do much good. Fischetti’s class was remedial. For students like you and I who had the opportunity to love reading early on, Fishetti’s approach may not be the best. To help struggling students get over their obstacles and gain the confidence needed to have the kind of love of reading you describe, this approach might help. This is a point that should be clarified, and I plan to include in a follow-up.
—
“Anyway, I’m all in favor of bringing in brain research, but hasn’t the right-brained/left-brained labeling been debunked to some extent?”
Like most “debunked” neuroscience, a specific interpretation of it has been debunked, but not every aspect of the concept. Many researchers take such pride in their being up-to-date on studies that they, in their own turn, oversimplify their condemnations. That said, I agree with your next point.
—
“To me, the most exciting discovery from brain research is the principle of neuroplasticity, which should make everyone believe they can get better at doing practically anything.”
AGREE. I pointed Fischetti to Dweck’s Mindset work. She began reading and says she loves it. My guess is Dweck’s concepts offer the best explanation for these gains. The possible imperfections of the brain theories she teaches may be practically irrelevant compared to the effect of students pondering their own ability to grow.
—
“I’m just very skeptical of claims like this. I’d be more impressed if a teacher reported that her students can’t get enough of school. The more programmatic the teaching becomes, the less that’s likely to happen.”
I agree. I hope teachers take Fischetti’s story as a data point to incorporate into their approaches, not as a programmatic script.
—
“Also, I don’t think you should publish articles like that one. If you have no idea what’s responsible for the teacher’s success.”
I DO have an idea of what’s responsible, and it comes directly from the teacher who is closest to the situation. Teachers aren’t stupid. If a class is getting impressive results, the first person to ask is the teacher. She’ll often see things your best statisticians can’t control for.
And I think good teachers don’t share enough (they’re too busy helping their students). We need MORE sharing and publishing of these types of articles about what good teachers do, not less.
—
“If you’re serious about finding out what’s going on, get some of your funders to sponsor controlled studies by qualified researchers.”
The problem with waiting for controlled studies (which EdSurge’s funders do fund a lot–probably too much imo) is a) they often try to isolate the effect of a product, but a product in isolation will always stink compared to a teacher who uses a product well to complement her own approach. The true cause of good outcomes usually lies in the less-isolatable arena of a teachers’ personality, drive, relationship with students, and pedagogy. I believe my approach of studying what measurably great teachers do will get us to insights faster than controlled studies. Which brings us to b) time: controlled studies can take 5+ years, and the if controlled variable is an edtech product, it will likely be obsolete by the time the study is completed. c) Controlled studies usually say nothing actionable for teachers. Their findings point to correlations that are every bit as misleading as the edfads that get pushed on teachers from above every few years. Given EdSurge’s tech-savvy readership, I think articles that temper tech enthusiasm by focusing on a teachers’ implementation rather than products will have a positive impact. Most edtech should go away, and fast. To discover how to focus the edtech industry’s efforts most helpfully, we should heed the instincts of great teachers.
LikeLike
Matt Bowman:
Dang it. You went ahead and quoted me. I’m sorry I took the bait. Next time I’ll be less thoughtful!
For the record, nobody said the teacher’s classroom was computerized. And who said teachers are stupid? I like the idea of teachers sharing success stories. I taught for a lot of years, and my instinct in reading the article was that it was both over-hyped and a little confusing. To me it came across not so much as a teacher success story, but as a moderately strong, albeit embedded, product endorsement that the maker might be able to use in marketing and sales efforts. It’s the over-hyping, in my view, that makes the article suspect. Because of your funding sources, I think your organization is going to have a tough time presenting clearly unbiased information about education. To use an EdSurge Newsletter term, the “KA’CHING” factor muddies the waters. As I suggested earlier, I thought the title was a little bit ridiculous.
The only thing I believe you’re flat wrong about is this:
“Giving books to students who are already demoralized about reading because they’ve always struggled won’t do much good.”
I’m guessing people who say things like that have never read Daniel Fader’s Hooked on Books (which apparently is out of print and out of favor). Here are two websites that might be of help: http://www.alan-ya.org/ and http://www.voyamagazine.com/ . Again, I’m a big fan of the Krashen-Parton approach.
There was a great panel on the Diane Rehm show today discussing the topic of brain-training games. (I’ve spent a certain amount of time on one of the popular websites.) One of the guests, who is involved with one of the competing sites, said his company’s research shows that software in which students are presented with cognitive tasks that grow incrementally more challenging can improve cognitive functioning, not just on those particular tasks, but generally. It’s worth listening to the whole hour: http://thedianerehmshow.org/shows/2014-03-24/growing-popularity-online-brain-training-games .
Just so you know, if you do end up doing a follow-up to that article, you don’t have my permission to quote me. Meanwhile, I’m thinking of retiring early from blog commentary and going into ed tech. It’s possible that my weak schmoozing skills will be a drawback, though.
LikeLike