Gary Rubinstein, the brilliant math teacher at Stuyvesant High School in New York City, has done it again.
He has dissected the Gates MET study–the one that says test scores are better at determining teacher quality than observations–and he says that the data in the study don’t make the point that has been widely reported.
Gene Glass, Research Professor at the University of Colorado, takes apart the MET study, and like Gary, says that the $50 million was a waste.
I wonder when Gates will abandon his mission to find the perfect metric to measure teacher quality. It isn’t working anywhere; it has perverse incentives; it is inaccurate and unreliable. How long will he stick with this failed idea?
Just think how many musical instruments that $50 million would have bought, how many librarians could have been rehired, how many after-school programs might have been funded.
Diane Ravitch's blog 
Why is the very well established fact that teaching operates via intrinsic motivation rather than external measures not known by those with money for more research or who make education decisions? I have studied a dozen or so books on education and the consensus on many aspects of education is clear yet rarely seen in practice.
It is all very sad that the same mistaken viewpoints are perpetuated. And a waste of money as you rightly point out.
50 million is chump change for Billy the Goates. And it’s not a waste for him as he gets more and more “free” publicity.
I too have been in the act of research for the last 3 or so years. Most research and literature I read classifies teaching as an “art”.
Gates et al. box teaching up into components that fulfill a circular reasoning that good teaching produces good test scores. They envision effective teachers as laboring to increase test scores.
Effective teaching is much like being an effective parent. You can try your hardest, yet children may not respond. Every parent and teacher knows children are different – they learn different, have different personalities and curiosities. School-life should be about giving the students the opportunity to question the world around them.
Also, in terms of parenting, it’s hard to classify one son/daughter as better than the others! Is it the one who makes more money? Is it one that attempts to positively affect others? It is one that is employed/unemployed, one that doesn’t hassle?
Teaching shares that commonality, in terms of parenting, and student/child output, with parenting – its hard to really know what works, when it works, why it works, and how to measure success vs. failure in terms of the actions of parents/teachers.
Excellent comparison and comment, ME. Thanks!
You keep calling it a waste. Which only goes to show that you don’t understand business — well, not Bill Gates’s brand of business.
Sometimes you spend money to improve your product.
Sometimes you spend money to destroy your competition.
Your idea of choosing wisely is not Bill Gates’s idea of getting his way.
Exactly.
Yep!
Or, 50 million could have built two new charter schools in Bellevue or Medina, like any normal rich philanthropist with an agenda would do.
Those Gates-funded studies are political documents, so if they move the political debate in his favor – highly likely, given the media echo chamber he enjoys – then he is probably satisfied with his return on investment.
Interestingly, AEI’s Rick Hess, normally a reform enthusiast, is also pretty critical of the MET study, not for its statistical ham-handedness, but for a more philosophical reason. He writes:
“I do think it’s a mistake to imagine that ability to move reading and math scores is universally a compelling proxy for being a ‘good’ teacher. And when we calibrate all of our other instruments based on their ability to predict value-added gains on reading and math assessments, we build our entire edifice of teacher quality on what strikes me as a narrow and potentially rickety foundation. When we see policymakers mandate teacher evaluation systems that rely almost wholly on observation and value-added, and feel comfortable in doing so because of the MET findings, I fear we’re getting way ahead of ourselves.”
Link: http://bit.ly/XE937b
This is the same reasoning Bruce Baker makes against MET. I believe it is a scientifically accurate assessment – how can we hypothesize that student surveys and administrator observations gauge effective teachers reliably when our proxy (VAM) is something we fully assume measures learning?
Tried to comment on Gene Glass’s blog but couldn’t get it to go through. So I’ll post it here.
Gene,
Glad to have found your blog-found it through D. Ravitch’s blog! Will have to go back through the prior postings. Thanks for this critique as it adds another arrow to the quiver to counteract the edudeformers nonsense.
But we don’t really don’t need that antiquated weapons technology (not really that antiquated if you’ve ever shot a compound bow). What’s interesting in using the arrow in quiver analogy is that in the EPAA archives there is a study by Noel Wilson who uses the multiple reentry nuclear analogy to describe his total destruction of educational standards and standardized testing through the 13 sources of error that he identifies, any one of which renders the process invalid which in turn render any conclusions drawn, as he says, “vain and illusory”.
Do you know of any rebuttal to his “Educational Standards and the Problem of Error” (EPAA-1998). I consider this the most damning of all the studies dealing standardized testing and the current obsession with it. I have looked for over a decade now for any rebuttal and the most I’ve found was a comment from another blogger that the study is “just another post-modern diatribe”. Not much of a critique. If you know of any rebuttals please list them.
Thanks,
Duane E. Swacker
This is no different than Gates pushing “Small Schools” for 10 years and then suddenly dumping it as it was another of their faulty programs. In fact, L.A. Mayor Villaraigosa, King Tony, and his schools PLAS, used small schools at Roosevelt High School. Right now they are admitting that small schools do not work and are apologizing to the community right now. The renewal of their charter is up for renewal in June. PLAS has broken almost every part of their MOU and Matrix which is required to continue to operate that school. Boyle Heights wants their schools back as LAUSD has failed and the so called “Education Mayor”, King Tony Villaraigosa, is trying to make a national name on being the “Education Mayor” when he and his program is a total failure. When you are a failure and admit it how are you to be the shining light of education? And should you be allowed to continue when you failed?
Yet small schools, well at least, small class sizes does work – maybe for reasons that VAM can’t measure.
George,
The Gates push for small schools was BASED ON DATA. Unfortunately for many, many, many students and taxpayers, Boy Gates did not understand the DATA he was given. He saw that “small schools” were over represented in the listing of ” effective schools” . He then DECIDED to push a small school initiative. It failed and he abandon the effort. “Dozens and dozens” of districts are still DEALING with and PAYING for his misunderstanding of the original DATA. If he knew how to interpret data, he would have noticed that large schools were also over represented in the listings of ” effective schools!” . Boy Gates wasted a BILLION of his and taxpayer money because he did not understand how Variation IS NOT distributed equally across distribution ( see “regression toward the mean”).
Howard Wainer and Daniel Kahneman have both explained his error in detail.
Boy Gates hired Kane because he didn’t want to make the same error again. He didn’t. But, the MET study is laughable. Kane is an intellectual embarrassment to Harvard.
Gate’s money and self confidence will never compensate for a lack of understanding of education, or, as it seems, simple stats. Surely Gates could also realise that there can be more small schools as the pupil count can be spread across more small schools than large schools. But how can he embark on such an expensive campaign on such just one factor? Education effectiveness is always multi-factorial.
LIke the Boy Gate moniker!
“Just think how many musical instruments that $50 million would have bought, how many librarians could have been rehired, how many after-school programs might have been funded.”
The $50 million was Gates’ own money, to invest as he chose.
Read between the lines.
There are at least two separate concepts here — 1) Evaluating teaching quality along a continuum from great to awful (like grading teachers A through F); and 2) Identifying teachers who are performing poorly (like grading teachers Pass/Fail).
The public generally does not care about drawing fine distinctions among competently-performing teachers — that is, identifying who is an A- teacher and who is a C+ teacher. And, teachers themselves are not calling for the drawing of such distinctions. Rather, the great public demand is to identify/remove the poorly performing teachers.
For these reasons, Mr. Gates — and we — should be focusing reform efforts on teacher evalaution approaches that identify/remove the poorly-performing teachers.
Montgomery County, MD (a large suburban county outside Washington, DC) has used a peer-evaluation approach (called “PAR”) for 10 years. Under PAR, over 500 teachers have been discharged or resigned-in-lieu-of-review. There have been few challenges to the discharges. The teachers union supports PAR. The teachers generally view PAR as being fair. There is no high-stakes testing with its serious adverse side effects. And, because the ultimate discharge decision is taken out of the hands of a teacher’s principal, PAR largely insulates the good teacher from the biased or vindictive principal. PAR is not perfect; it could be tweaked a bit (particularly to allow initiating a review more quickly). But it’s a huge improvement on both the traditional principal-observes-and-evaluates model and the high-stakes-testing/value-added model.
The NY Times has written favorably about PAR. And, even Arne Duncan has noted it, albeit in passing.
Mr. Gates would have accomplished a great deal more if, instead wasting $50M on the MET study, he had spent a few million studying and publicizing Montgomery County’s PAR program — an approach to identifying/removing poorly-performing teachers that has a proven track record of removing teachers with no adverse side effects.
Good analysis. Teachers do not like fine distinctions either because we see each other as all ‘hard-workers’. Rarely do I view a colleague as lazy and incompetent, other than a handful in 15 years of teaching.
Can you imagine the sample size I have to draw from? Four different schools, two different states, hundreds of teachers (many of whom I mentored and observed in-class), and rarely have I seen a teacher that was not doing everything possible to enhance the teaching/learning environment.
I’ve gone into a lot of classrooms too and can’t really remember anyone who I thought should be fired.
My children have run across several and at least one who should have been put on medical leave.
I think that peer review is exactly the right way to go.
@teachingeconomist
Kids have a vested interest in portraying their teacher as incompetent. Their parents are more likely to side with their kids against an “incompetent” teacher then a “competent” teacher.
No doubt that is sometimes true, but not in these cases.
My sentiments, exactly!!!!!!!
“Bill Gates:Experienced Educator!($50 Million Dollars Doesn’t Lie)!”
http://oldschoolteach.blogspot.com/2013/01/bill-gatesexperienced-educator50.html
Kind of a drastic shift from having the principle 100% responsible for teacher evaluation down to 0. How much will the new evaluation system cost the state?
Will this be used to unfairly terminate older teachers and steal their retirement money in a pension system that is weighted to working to a set age?
I wouldn’t disagree that the evaluation system, pay system, and pension system need to be made fairer, but poorly planned drastic changes could do more damage than good.
Personally I believe seniority increases should stop after 10 years. How much is a masters, a doctorate, or a board certification worth? Maybe those rates need to be adjusted, but certainly not ignored.
Last, the three people doing the teacher evaluation need to be board certified teachers with years of experience. If it is twenty year olds with a traing course, that is the equivalent of having med students do the evaluation of veteran doctors or surgeons.
Are you saying State Board of Ed certified or National Board Certification?
If your saying basic certification, then agreed.
If it is national board certification, are you saying that paying a few thousand dollars and going through a Pearson evaluation system makes you so much of a better teacher that only those folks could successfully evaluate peers? This I disagree with strongly.
Wilbert,
Agree with your statement about “national board” certified. Pure bunk in my mind.
Duane
“How much is a masters, a doctorate,…” To schools of education, the salary increases that are automatically associated with those degrees are pure gold.
Well, my main point is, if one does not like the Gates study, provide a better alternative for teacher evaluation that most teachers agree with.
And second, the compensation system is relevent. This is presumably the endgame of the Gates study. Are there any improvements that can be made on the current structure to make it fairer? Is a compensation system from the 1950s applicable to today’s flexible world? Do the rules allow you to travel to another state or another profession and take your retirement with you?
Reblogged this on Transparent Christina.
Gate$, and his Chief Lapdog$, will never stop doing things which blame the working stiffs. Period.
What is their choice? Look at the system and recognize that those in charge aren’t accountable or responsible? Recognize that the system is set up to optimize rewards for the Boy$ at the top while dumping all accountability downhill ?
If Gate$ and his doublethinking Chief Lapdog$ focused on making the system work better, they’d have to devise metrics for judging managers – other than ‘the Big Shot down the street just got a 5 million or 1/2 million dollar raise, so, I better get 1 too!’ metric of USA business over the last 30+ years.
Oh yeah – about that management accountability stuff – how could they take $600,000,000 of annual public money for public investement and funnel it through scores of Rhee-Kipp-Kopp Krime syndicates if there is accountability at the top??
Gate$ and his Chief Lapdog$ will always be about blaming the serfs – their lies will get warmer and fuzzer as time demands, they’ll get more sopisticated as times evolve, but, you want to understand where they’re coming from?
Google “Lee Atewater __gger __gger __gger” (the n word) , and think of non rasicst elitist lying thieves. Period.
Only engaging those people on a policy level plays exactly into their hands.
rmm.
ooops! that is $600,000,000,000 of public spend !
One has to understand the epistemology of Education Capitalizers and Knowledge Commodifiers. They hate the very idea that knowledge resides in human beings and not in their servers. They hate it even more that it resides in human beings who practically give it away, asking for nothing more than a moderate middle class existence and a modicum of security in their old age. They hate it that they can’t cap the wells of knowledge and hold the world hostage until the price suits their dreams of avarice, when and only when they’ll deign to send a few dribs and drabs down the pipelines under their control. The only solution, as they see it, is to convert the human cachers and conveyors of knowledge into just another bank of servers they own.
Can you let us know who the great “they” are that hate all this? “They” are totally screwed in any case, as a student can learn far far more with a click of a button than you or I could ever have learned in the high schools we attended.
Well, you’ve already convinced me you are not an economist — and now you convince me you are not a teacher.
Not really an answer to my question, but I was not really expecting one.
The information revolution has freed knowledge from the physical confines of the classroom. With a book, access to Wikipedia, and the web site http://math.stackexchange.com/ , my son was able to bypass the gatekeepers of knowledge in his high school and teach himself enough abstract algebra to skip the first year and a half of the graduate algebra sequence at an AAU university while still in high school.
The world is exactly the opposite of what you discribe. The knowledge gatekeepers, who ruled the education world when you and I were in school, have no power anymore. Knowledge is free for the taking.
Actually, I have to agree with teachingeconomist here. I’ve been saying much the same thing for years now. This is part of the reason why some professors fly into conniptions and foam at the mouth whenever anyone mentions wikipedia. “WIKIPEDIA!!!”
Yes, there are reasons why wikipedia isn’t the best source of information, but if you ask me there’s something more to their paroxysms of fury than that.
I have to depend on my son’s opinion here, but he claims that he has never found a significant error in the mathematics on Wikipedia.
I gave a generic description in Capitalized tersm of the estate I had in mind — I didn’t want to be seen as picking on Mr. GatesKeeper himself, as we all know he’s suffered enough.
As for your giddy e-thusiasm, you are expressing a worldview much like the one I myself once maintained, roughly between 1985 and 2005, and I’m sure I put all the same sentiments in print or e-print at one time or another in those e-halcyon years.
But that was WayBack when I naively believed that the e-conomics of IT would e-ventually free us all from the relations of power in whose thrall that old-fangled economics we knew so well had formerly enslaved the masses.
Sad to say, but wiser, I have learned better since then …
I wasted a good five years of my life on Wikipedia before moving on to the greater weevils that assail us today, but here is one of the still active watchdog forums for those of you who remain as yet clueless about the lesser weevils of our nature, education, and public information.
Wikipediocracy ☠ The Wikipedia Forum