You know that “value-added modeling” (VAM) has failed everywhere. Several courts have blocked its use. It is one of those zombie ideas that never works and never dies. Actually, it will die because some states have dropped it as an expensive and useless exercise that does not identify the best or the worst teachers. The Gates Foundation pushed it but the Gates-funded evaluation showed that after $575 million spent, it did not improve student test scores, it did not identify teacher quality, it nearly bankrupted those that tried it.
Steven Singer has saved us all some time by listing the 10 Top Reasons it doesn’t work.
Diane Ravitch's blog 
Value-added measures have been thoroughly discredited. But they are still used in Ohio and other states. Big money is part of this charade.
I love this one page brief with refers to and eliminates the “hedges” in statements from other authoritative sources:
VAMs Are Never “Accurate, Reliable, and Valid” http://journals.sagepub.com/doi/pdf/10.3102/0013189X16651081
LikeLike
VAM and SLO remain in widespread use because legislatures mandated them at the insistence of Arne Duncan, so as to be eligible for Race to the Top money.
LikeLike
The hedging is the main reason that VAM has lasted as long as it has.
For example, rather than state quite unambiguously that VAM should never be used to evaluate individual teachers for high stakes purposes, the American Statistical Association left the door oppen enough for some to claim that they do not oppose VAM.
And statistics departments at top universities like Harvard have ignored VAM entirely, even though economists at their universities (like Raj Chetty) were making nonsense statistical claims about VAM.
Had the statisticians at Harvard critiqued Chetty’s study, they could have nipped it in the bud before it was touted by Obama in his SOTU speech.
LikeLike
ASA said it should never be used to evaluate individual teachers, but left the door open to say that it might be used for entire districts.
Chetty told me in a personal email that he expected to win the Nobel Prize for his work on VAM. He hasn’t yet.
LikeLike
ASA implied that it should not be used to evaluate individual teachers but they never stated quite unequivocally that
“VAM should never be used to evaluate individual teachers for high stakes purposes”
Implying and stating unambiguously are very different.
The wording matters because when you do not state things unambiguously, people like Chetty can claim that you meant something different from what you actually did.
Even leaving the door open a crack is too much.
RE Chetty, he is a joke and so is the so called Nobel prize in Economics.
It’s not a real Nobel prize and quite frankly, Chetty would fit right in with some of the other clowns who have received it.
LikeLike
Chetty has a very poor understanding of statistics.
No real scientist would ever take the low correlations involved with VAM (0.24) seriously (especially since they may not even indicate a causal relationship), but that teachers have been publicly shamed (eg in the pages of the LA Times) and lost their livelihoods based on what amounts to junk science is actually nothing less than criminal.
I am not a statistician (I have a bachelor’s in physics),but from what I know about statistics and from a reading of the critiques of Chetty et al (eg, by Moshe Adler), I have been convinced beyond any reasonable doubt that these people were up to no good from a statistical standpoint.
Chetty et al are obviously very confused about some very basic statistical principles. If the signal does not rise above the noise at age 30 for their original dataset, according to Chetty et al’s own admission (“one cannot reject the hypothesis that the effect [at 30] is zero”), how can they claim, as they did (in the very same paragraph!) that “impact [at age 30] is actually larger” [than at 28]?
They can’t. That’s nonsense.
But in my opinion, there is something much more insidious than simple ignorance involved. Chetty et al rather conveniently avoided addressing some of the critical issues that Moshe Adler brought up, particularly his point about using the latest, greatly augmented data set to calculate an estimated income increase for 30 year olds for their second paper.
Why wouldn’t they include the age 30 result, when, as Moshe Adler pointed out, they certainly had sufficient data (even if they did not before)? If they still find a non-significant result at age 30, even with the much larger data set, that information is highly relevant and absolutely should have been included in their last paper.
But of course, then they would have had no paper.
The very idea that Chetty et al can detect (divine?) such a small differential “impact” (~$200 at age 28) from the teaching of a single teacher decades before stretches the outer bounds of credulity, especially given the noise due to all the other factors, but that Chetty et al extrapolated that effect for decades thereafter is just plain ridiculous.
That’s not legitimate use of mathematics. It’s mathturbation worthy of the junk heap, certainly not a prize.
That the work (junk) of folks like Chetty et al has been used to justify abusing and even firing some of America’s most dedicated and critically important professionals is nothing less than grotesque, sickening and immoral.
LikeLike
Chetty seems to fashion himself as a latter day Ramanujan.
What a joke.
LikeLike
Re, so called “Nobel economics prize”
See
“the economics Nobel isn’t really a Nobel”
https://fivethirtyeight.com/features/the-economics-nobel-isnt-really-a-nobel/
And
There is no Noble Prize in economics
https://www.alternet.org/economy/there-no-nobel-prize-economics
Its a fake Nobel Prize. A fraud, not unlike some of those who have received it.
LikeLike
People should be very careful before they refer to the econ prize as a “Nobel Prize”
As the Alternet piece makes clear, it has a sordid history, being born as a marketing ploy intended to bolster free market ideology.
“What started as a project to help the Bank of Sweden achieve political independence, ended up boosting the credibility of the most regressive strains of free-market economics, and paving the way for widespread acceptance of libertarian ideology.”
“Take [Friedrich] Hayek [first Libertarian to win prize]: Before he won the award, it looked like Hayek was washed up. His career as an economist was essentially over. He was considered a quack and fraud by contemporary economists, he had spent the 50s and 60s in academic obscurity, preaching the gospel of free markets and economic darwinism while on the payroll of ultra-rightwing American billionaires. Hayek had powerful backers, but was out on the fringes of academic credibility.
But that all changed as soon as he won the prize in 1974. All of a sudden his ideas were being talked about. Hayek was a celebrity. He appeared as a star guest on NBC’s Meet the Press, newspapers across the country printed his photographs and treated his economic mumblings about the need to have high unemployment in order to pay off past inflation sins as if they were divine revelations. His Road to Serfdom hit the best-seller list. Margret Thatcher was waving around his books in public, saying “this is what we believe.” He was back on top like never before, and it was all because of the fake Nobel Prize created by Sweden’s Central Bank.
Billionaire Charles Koch brought Hayek out for an extended victory tour of the United States, and had Hayek spend the summer as a resident scholar at his Institute for Humane Studies. Charles, a shrewd businessman, quickly put the old man to good use, tapping Hayek’s mainstream cred to set up and underwrite Cato Institute in 1974 (it was called the Charles Koch Foundation until 1977), a libertarian thinktank based on Hayek’s ideas. Even today, Cato Institute pays homage to the Swedish Central Bank Prize’s role in the mainstreaming of Hayek’s ideas and Hayek’s influence on the outfit”
LikeLike
I read almost two fifths of the Harvard Classics this summer, but have had to put the reading on hold lately while getting ready for school. So I don’t have quotes of the day from the Five Foot Shelf of books right now, but this one from Diane will definitely suffice as the quote of the day: “Chetty told me in a personal email that he expected to win the Nobel Prize for his work on VAM. He hasn’t yet.” Ha! That’s very funny. Reminds me of the time Guggenheim expected to win an Oscar after staging scenes in his “documentary” Waiting for Superman. We’re all still waiting for corporate management schemes to be super…. Still waiting.
LikeLike
Everything the Reformers do has failed. A perfect record.
LikeLike
“That the work (junk) of folks like Chetty et al has been used to justify abusing and even firing some of America’s most dedicated and critically important professionals is nothing less than grotesque, sickening and immoral.”
Exactly!
Add to that the abuses that are heaped upon the most innocent of society, the students, in the error and falsehood-filled, psychometrically fudged standards and testing regime malpractices (including SLOs) and one can only come to the conclusion that our public education laws and mandates and practices are not only invalid and absurd but, sadly, unethical and also immoral.
Who can support such insanity?
LikeLike
Duane
In my opinion, part of the responsibility falls on colleagues who were silent.
In the case of Chetty, that includes people in the statistics department at Harvard and now Stanford.
If they had taken the time to critique what he was claiming, they could have made a big difference before the damage was done.
There best way to influence an academic is to demonstrate that they are wrong and/or demonstrate that they don’t understand what they are doing.
The latter — embarrassment by colleagues — is probably most the effective means of influence.
LikeLike
Waiting for Superman. We’re all still waiting for corporate management schemes to be super…. Still waiting.
Ha ha ha!
“Waiting for Success”
Waiting for “Success”
Like waiting for Godot
Deformers never rest
But Failure is in tow
LikeLike
“Waiting for Success”, verse 2
Godot, he never comes
“Success” does not appear
To all Deformer bums:
“The Superman ain’t here”
LikeLike
Sincerest apologies to Samuel Beckett, of course.
LikeLike
SDP,
Thanks for sharing the Adler review. How damning is this?
VI. Usefulness of the Report for Guidance of Policy and Practice
The two-part report under review here uses questionable techniques to reach conclusions that are not supported by the data. These problems render the report of no value in guiding educational policy.
LikeLike
Meant to add: There is also no research to support the use of SLOs as a proxy for VAMs. An SLO is a convoluted writing assignment for the many teachers, estimated to be about 69%, who do not have job assignments attached to statewide tests. SLOs were marketed by William SLotnick, beginning in 1999 as part of a Denver pay for performance scheme. I have references if someone asks.
LikeLike
Please post a link, Laura! Gracias in advance!
LikeLike
You will find a link to the complete paper with references here. You can download it.
http://vamboozled.com/laura-chapman-slos-continued/
LikeLiked by 1 person
Thanks again, Laura!!
LikeLike
Laura
Here is a “field sample” SLO for grade 7 Art (New York State).
Read it and weep.
file:///C:/Users/bobrick/AppData/Local/Packages/Microsoft.MicrosoftEdge_8wekyb3d8bbwe/TempState/Downloads/art-grade_7%20(3).pdf
LikeLike
Try again:
https://www.engageny.org/resource/student-learning-objectives
LikeLike
Is that sanity I hear?
VAM never passed the laugh and then gasp test among intelligent people.
LikeLike
Most rounded and intelligent people, that is. One thing about this whole mess with the deformers and privatizers is learning about how so many clearly intelligent people can seemingly fully support and meticulously defend clearly stupid ideas. We even see it with the upper-echelon Treasonous people.
Some on this blog have discussed smart yet dumb people, and there are so many dimensions to reasoning and judgment, and deeply understanding ourselves and others within all of the various societal and physical laws that govern us, at least in a worldly sense. Ironically, it was Obama who admitted to blind spots, that we all have them. We all most likely have local and generalized myopias as well, probably thanks in equal parts to wiring and experience. Luckily, though, our potential is so limitless, at least beyond what we can reasonably predict and imagine, and that is also the potential of our children.
LikeLike
If student test scores are not an adequate metric to determine teacher performance, then what is? (I concur that evaluating teachers solely on test scores, is neither fair nor accurate).
In a decade, only 19 teachers were terminated in California, based on poor performance. see
https://townhall.com/tipsheet/cortneyobrien/2014/04/21/number-of-tenured-teachers-in-ca-fired-for-poor-performance-in-last-decade-19-n1826813
What criteria should be used to determine if a teacher is not delivering? And should it not be easier to terminate a teacher, who is not performing?
LikeLike
There is a national teacher shortage. This is not a good time to hunt down bad teachers. Almost 40% of Teachers quit within five years of starting. The kids get rid of those who can’t take the heat.Are you prepared to become a teacher? Why not?
LikeLike
I concur that there is a shortage. When is a good time to “hunt down” bad teachers? Does our society really want bad teachers in the classrooms of America?
With the low wages and poor treatment of teachers, I am not surprised that so many teachers resign. Stop the presses.
Are you serious that the “kids” get rid of teachers? How can they do that? Are “kids” able to fire incompetent teachers?
I am an engineer, not a teacher .I am too old to go back to school ,and attain certification.
I want to earn a salary and benefits adequate to support my family. I am not prepared to take a vow of poverty, and become a classroom teacher.
LikeLike
Charles,
Anyone with the appropriate training and experience should be thanked and respected. When is the right time to hunt down and purge “bad” engineers? After the plane crashes or the bridge falls down, or sooner?
LikeLike
Q When is the right time to hunt down and purge “bad” engineers? END Q
If an engineer is not delivering services to his/her employer, to the satisfaction of the employer, then the person should be terminated. It is far easier to terminate a person in the private sector, than in the public sector.
Virginia (and most other states) have “at-will” laws, for employment. An employer may terminate an employee at any time, and for any reason or for no reason. These laws benefit employees.
Employers are much more likely to hire a person, when they are free to fire a person.
LikeLike
Try teaching 200-250 children a day for a salary of $36,000 a year and tell me who has the harder job.
LikeLike
The way to manage teachers is to be a principal who, through experience, understands how teaching is accomplished, and for the administrator to get out from behind the computer screen and interact with the students and teachers, to be supportive and helpful, to ask people where problems exist and ask for their input in solving the problems. When an organization uses data to stacked rank employees instead of utilizing superior human management, the organization flops, as shown by Microsoft: https://dianeravitch.net/2012/07/07/what-the-microsoft-culture-is-doing-to-education/
LikeLike
Just a basic fact: government contractors are not the same as the “private sector.”
Specifically, government contract workers are not “at will employees”. In fact, government contract workers enjoy many job protections (including protections against at will firing) that truly “private sector” employees do not have.
Don’t believe me?
Look it up.
LikeLike
@SomeDAMpoet: I am a contract employee, on a government project. I have done contract work for many years, both here in the USA and abroad.
Contract workers do NOT enjoy civil-service protections. I can be terminated at any time, for any reason, or for no reason. My employment and nearly all federal contract employment is “at -will”. (To be fair, I can resign at any time, for any reason, or for no reason).
LikeLike
The study finds that “about 98 percent of teachers manage to attain tenure—or, more accurately, “permanence.”
What? Has something seriously changed. These figures can’t be accurate. Of the four teachers who were hired at my local school last year, only one remains. I would say that the number should be closer to 60 or 70. And then I realized that this study is about five years old, too.
Secondly, the whole “in a decade” phrase isn’t exactly accurate either. Since the study was in 2014 and not many districts participated in VAM or its like, it is rather erroreous to connect that to the whole teacher population of CA. At most 10 to 20 percent of teachers in CA were under the VAM spotlight, so you’ll need to lower that 300,000 number to at least 60,000 teachers–and that’s very generous.
LikeLike
“In a decade, only 19 teachers were terminated in California, based on poor performance.” End quote by C.
Only 19 teachers were terminated in 10 years? So what, what does that prove one way or the other. Would 1,000 teachers being fired be better? Many teachers quit before they can even be fired for whatever reason, many teachers take early retirement because they are burned out by all the mandates imposed on the real public schools. This meme that “too few teachers are being fired” is a typical ploy of the anti-union and pro charter/reform crowd. They love the idea of mass firings of teachers, it gives them a warm inner glow. Ugh.
LikeLike
I think that 19 teachers being terminated for non-performance in the state of California in 10 years, is a very small number. This should show anyone, that it is very difficult to terminate an incompetent teacher in the state of California. (There may be other factors involved, as well.).
I think we can all agree, that one incompetent/ineffective/ unsatisfactory teacher in a classroom, is one too many. There is no one specific number. The reasons why teachers voluntarily leave (resignation/retirement), vary with the individual.
We should agree that there should be an effective, non-prejudicial, accurate and objective methodology to evaluate teachers. (Just like any other profession). If a teacher is not meeting standard, then the individual should be removed from the classroom.
LikeLike
Charles,
Sorry I don’t agree. When you say “all can agree,” count me out. The number of people entering teaching has dropped dramatically in California. There are shortages in many districts. As I explained to you, nearly half of those who start teaching leave.
And you want to fire teachers?
How do you feel about hiring young kids with only five weeks training?
Would you hire an engineer who never went to engineering school but had five weeks of training?
Charles, now is a time to cherish the teachers with experience and dedication and cut the crap about hunting for bad ones. If they are bad, their principals can fire them before they get tenure.
LikeLike
Q Sorry I don’t agree. When you say “all can agree,” count me out END Q I do not understand you. Are you disagreeing, that incompetent and poorly performing teachers should not be terminated, and removed from the classroom? Am I understanding you right?
Q The number of people entering teaching has dropped dramatically in California. There are shortages in many districts. As I explained to you, nearly half of those who start teaching leave.
And you want to fire teachers? END Q
The recent spate of strikes in Oklahoma, Kentucky, West Virginia, etc underscores the fact that teacher salaries are inadequate ( at least in the aforementioned states). California has a high cost of living, and (teacher) salaries there have not kept up, either. The result is fewer qualified applicants entering the profession. There are shortages.
I do want to fire incompetent and poorly-performing teachers, yes. What does one have to do with the other. Salaries are inadequate, no dispute.
Q How do you feel about hiring young kids with only five weeks training? END Q
What is this all about? It depends on what you are hiring the individuals to do. You can obtain a real-estate license in Virginia with two(2) weeks of training.
Q Would you hire an engineer who never went to engineering school but had five weeks of training? END Q
Again, what is this all about? Engineering is a serious profession, and it takes years of academic preparation.
Q now is a time to cherish the teachers with experience and dedication and cut the crap about hunting for bad ones END Q
I agree that excellent teachers should be compensated properly, and encouraged. I still maintain, that poorly-performing teachers, should be terminated, and removed from the classroom. How can you possibly advocate, keeping bad teachers on the public payroll, and in front of students? I do not get it.
Q If they are bad, their principals can fire them before they get tenure. END Q
In a perfect world, this should be true. In the USA, it is just a fantasy. Look at the statistics. In a decade, in the entire state of California, only 19 (nineteen) teachers were terminated for non-performance. Are you asserting that this number represents all of the non-performing public school teachers in the largest state in the union, in a decade?
Q The percentage of teachers dismissed for poor performance in Chicago between 2005 and 2008 (the most recent figures available) was 0.1 percent. In Akron, Ohio, zero percent. In Toledo, 0.01 percent. In Denver, zero percent. In no other socially significant profession are the workers so insulated from accountability END Q
see
https://www.newsweek.com/why-we-must-fire-bad-teachers-69467
and
http://www.aei.org/publication/why-dont-teachers-get-fired-for-poor-teaching/
LikeLike
Charles,
The data you cite show that there are very few “bad” teachers, not that they are being hidden and protected.
My question clearly related to teaching, not to getting a real estate license. How do you feel about hiring “teachers” with only five weeks of training to teach in very tough neighborhoods? How do you feel about hiring “engineers” who have only five weeks of preparation?
Teachers can be fired at any time for any reason when they are not tenured. If administrators are giving tenure to “bad” teachers, we have an administrator problem.
Note: Teachers do not give tenure to themselves.
I know one of the authors of that Newsweek article you cite. She told me that she wished she had never written it.
You should not make your ignorance of teaching and education public. It is embarrassing.
LikeLike
The Newsweek article lists only one(1) author. Evan Thomas see https://en.wikipedia.org/wiki/Evan_Thomas
If a woman told you she was sad that she wrote that article, I believe you may be confused about the article and/or the author.
LikeLike
The other author was Pat Wingert. Obviously she asked to have her name removed. I read the article when it first appeared, and footnoted it accordingly.
If you google the original article, you will see that it was co-written by Evan Thomas and Pat Wingert. Wingert was a veteran writer at Newsweek.
https://fair.org/home/newsweek-wants-accountability-for-teachers-not-editors/
http://www.ascd.org/publications/educational-leadership/may10/vol67/num08/What-Newsweek-Gets-Wrong.aspx
https://kansaseducation.wordpress.com/2010/03/09/newsweek-why-we-must-fire-bad-teachers/
http://blogs.edweek.org/edweek/Bridging-Differences/2010/03/the_shame_of_newsweek.html
Pat Wingert won a Spencer Fellowship where she learned more about education and came to regret that article.
You are ignorant about education, you are unwilling ever to learn anything new, and I don’t think you should comment here anymore.
LikeLike
I stand corrected. I only read the one article in Newsweek, and saw only one author. See, I can learn!
LikeLike
Q The data you cite show that there are very few “bad” teachers, not that they are being hidden and protected.
My question clearly related to teaching, not to getting a real estate license. How do you feel about hiring “teachers” with only five weeks of training to teach in very tough neighborhoods? How do you feel about hiring “engineers” who have only five weeks of preparation?
Teachers can be fired at any time for any reason when they are not tenured. If administrators are giving tenure to “bad” teachers, we have an administrator problem.
END Q
Data can be interpreted in many ways. The fact that very few teachers are terminated for non-performance, does not mean that there are very few bad teachers. Nonsense.
The fact that very few teachers are terminated for non-performance means that it is very hard to terminate bad teachers. And the problem extends even to our northern neighbor. Here is a fascinating article about the problems in firing non-performing teachers in Canada (and Virginia). see
https://www.macleans.ca/news/canada/why-its-so-hard-to-fire-bad-teachers/
Poorly-performing teachers are more often transferred to a different school, than they are terminated.
As to your other questions: I do not support hiring a person with only five weeks of training to be a school teacher. Ludicrous. Teaching is a serious profession, which requires many years of academic preparation.
Same with engineering. It takes many years of academic study to become an engineer.
As far as terminating non-tenured teachers, I am certain that it varies from state to state. Ideally, a school system should be able to remove a teacher when the individual is not performing up to standard.
The very small number of teachers, that are terminated for non-performance, should reveal to anyone, that it is very difficult to terminate a teacher when necessary.
LikeLike
At least we agree that Teach for America is a very bad idea.
There is a national teacher shortage, due to deeply flawed evaluations and low pay.
Do you have any positive suggestions to get more excellent teachers, or are you concentrated solely on firing teachers and worsening the shortages?
LikeLike
Q Do you have any positive suggestions to get more excellent teachers, or are you concentrated solely on firing teachers and worsening the shortages? END Q
Sure! I would like to see more highly-qualified and effective teachers in American classrooms. There are many good and cost-effective ways to do this.
In Kentucky, there is a program where the state will loan money to a college graduate to earn a master’s degree, with certain stipulations.
-The recipient must have an undergraduate degree from an accredited college.
-The recipient must obtain the master’s degree at an accredited institution.
-The recipient must commit to teach in a Kentucky public school for a minimum of three(3) years, and then the loan will be cancelled.
-The state will select the school that the grantee will teach in.
This is a great program, my sister got one of the loans, and went on to teach in a public school.
There needs to be alternate certification programs for individuals with advanced degrees in certain fields (STEM, and foreign languages). Virginia has a “career-switcher” program, which takes qualified professionals, and sends them to a state-operated educational certification program, to obtain their teaching credentials.
The Virginia Educational association, and nearly all of the public school teachers in Virginia, enthusiastically support this program.
Other states can pick up on it!
Public schools, should operate “internship” programs for college students, to introduce them to teaching careers. Qualified undergraduate students could teach in public schools, under the direction of an experienced teacher.
I am NOT concentrating only on firing unqualified and non-performing teachers. I continue to assert, that poorly-performing teachers should be removed from the classroom, and terminated.
There is indeed a shortage of qualified teachers (in some states), and that the salaries offered are not keeping up with the market. This underpayment only exacerbates the shortage. No one should have to take a vow of poverty to be a teacher.
Nevertheless, it is unfair to students, and not cost-effective to keep poorly performing teachers in the classroom.
LikeLike
Thanks, Charles. There is no crisis of bad teachers. That’s a myth and a fraud. Get down on your knees and thank God for the wonderful teachers who show up every day and teach our children, while paid meager salaries and subject to to budget cuts, insane state and federal mandates, and micromanagement by politicians. Show respect and appreciation.
LikeLike
“Crisis” is an over-used word. I support having highly-qualified, motivated, well-compensated teachers in our nation’s classrooms. Who can be opposed to that?
LikeLike
Charles,
Anyone who teaches is motivated and underpaid.
LikeLike
Evaluation systems apart from test scores exist now. That’s a whole other debate, perhaps involving outdated laundry lists or inappropriate rubrics verging on organizational mania inappropriately focusing on student response and conduct, and inappropriately trying to disaggregate pedagogical talents and habits, and as always confusing correlation and cause.
LikeLike
The Performance Standards Consortium of New York has developed an evaluation system that is working very well. Better student outcomes without taking standardized tests.
LikeLike
Hmm. No mention of Danielson.
http://www.performanceassessment.org/how-it-works/
LikeLike
If student test scores are not an adequate metric to determine teacher performance, then what is? ”
The question should simply be “what is a valid way of evaluating individual teachers?”
If VAM (or other methods based on student scores) is not valid for that purpose, that should be the end of the discussion.
Even if VAM were the only way to evaluate teachers (which it is not), that would not be a legitimate argument for it’s use.
Unfortunately, some have made precisely that argument, which is like saying that “All we have to build car bridges is popsicle sticks, so we should go ahead and build them with what we have”.
It’s a stupid argument, of course, but no stupider than “VAM is all we have so we should use VAM”, which is a logical fallacy.
Somewhat surprisingly, some fairly intelligent people (eg, ,Bill Gates) have made some very dumb arguments in the name of education “reform”
LikeLike
Delivering? What a strange view of teaching you have. Also are you aware that the ngram timeline for “teacher performance metrics” for teachers tracks the corporate view of students and teachers as little more than producers of test scores? Same for CEO performance.
LikeLike
If an administrator thinks that a teacher is a bad person, one who damages students by being cruel or socially inappropriate, this is one problem. If an administrator thinks the teacher should deliver lessons differently, that is quite another. No difference of agreement about content or delivery should be the basis for dismissal. Students need the experience of different approaches to the profession.
Political differences should never be the basis for teacher dismissal. Variance of political philosophy sounds like a prescription for a good school to me. Obviously, no religious test or general ethnic background should enter in.
So what is left? Testing the students is preposterous on the face of it, and now experience has proven the idea way more than intellectually bankrupt.
I once had a teacher who would teach clas for 15 or 20 minutes, then tell us to go to the library and read. Lazy bum, right? Not really. I took his advice and it led to my being interested in history. That freedom was my door to scholarship. The point is that you never know what a student needs. I have never done what that teacher did. Have I hurt someone? Did he?
Getting rid of bad teachers is way more complex than anyone will admit. Just like getting rid of bad employees of any kind that provide a service, getting rid of “bad” teachers can be fraught with unintended consequences.
LikeLike
From the article:
“6) VAM Scores are Based on Flawed Standardized Tests.
When you base teacher evaluations on student tests, at very least the student tests have to be valid. Otherwise, you’ll have unfairly assessed BOTH students ANDteachers. Unfortunately standardized tests are narrow, limited indicators of student learning. They leave out a wide range of important knowledge and skills leaving only the easiest-to-measure parts of math and English curriculum. Test scores are not universal, abstract measures of student learning. They greatly depend on a student’s class, race, disability status and knowledge of English. Researchers have been decrying this for decades – standardized tests often measure the life circumstances of the students not how well those students learn – and therefore by extension they cannot assess how well teachers teach.”
Singer repeats a couple of the false educational memes in #6. First, the standardardized tests aren’t just flawed, they are COMPLETELY INVALID as proven by Noel Wilson in his never refuted nor rebuted 1997 treatise “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700.
Second, there is no “measuring” of student learning by any stretch of the definition of that word. To continue to use that false language only serves to further embed the false discourse of “measuring” that lends a false, a pseudo-scientific sheen to assessment of the teaching and learning process.
The most misleading concept/term in education is “measuring student achievement” or “measuring student learning”. The concept has been misleading educators into deluding themselves that the teaching and learning process can be analyzed/assessed using “scientific” methods which are actually pseudo-scientific at best and at worst a complete bastardization of rationo-logical thinking and language usage.
There never has been and never will be any “measuring” of the teaching and learning process and what each individual student learns in their schooling. There is and always has been assessing, evaluating, judging of what students learn but never a true “measuring” of it.
But, but, but, you’re trying to tell me that the supposedly august and venerable APA, AERA and/or the NCME have been wrong for more than the last 50 years, disseminating falsehoods and chimeras??
Who are you to question the authorities in testing???
Yes, they have been wrong and I (and many others, Wilson, Hoffman etc. . . ) question those authorities and challenge them (or any of you other advocates of the malpractices that are standards and testing) to answer to the following onto-epistemological analysis:
The TESTS MEASURE NOTHING, quite literally when you realize what is actually happening with them. Richard Phelps, a staunch standardized test proponent (he has written at least two books defending the standardized testing malpractices) in the introduction to “Correcting Fallacies About Educational and Psychological Testing” unwittingly lets the cat out of the bag with this statement:
“Physical tests, such as those conducted by engineers, can be standardized, of course [why of course of course], but in this volume , we focus on the measurement of latent (i.e., nonobservable) mental, and not physical, traits.” [my addition]
Notice how he is trying to assert by proximity that educational standardized testing and the testing done by engineers are basically the same, in other words a “truly scientific endeavor”. The same by proximity is not a good rhetorical/debating technique.
Since there is no agreement on a standard unit of learning, there is no exemplar of that standard unit and there is no measuring device calibrated against said non-existent standard unit, how is it possible to “measure the nonobservable”?
THE TESTS MEASURE NOTHING for how is it possible to “measure” the nonobservable with a non-existing measuring device that is not calibrated against a non-existing standard unit of learning?????
PURE LOGICAL INSANITY!
The basic fallacy of this is the confusing and conflating metrological (metrology is the scientific study of measurement) measuring and measuring that connotes assessing, evaluating and judging. The two meanings are not the same and confusing and conflating them is a very easy way to make it appear that standards and standardized testing are “scientific endeavors”-objective and not subjective like assessing, evaluating and judging.
That supposedly objective results are used to justify discrimination against many students for their life circumstances and inherent intellectual traits.
LikeLike
Agreed.
LikeLiked by 1 person
Duane
The distinction that you highlight between “subjective” (assessing) and “objective” (measuring) is key.
Everyone who uses the term “measurement” or “metric” to describe educational assessment is confusing the two and thereby committing a serious error.
LikeLike
SDP,
You are correct in your second statement about committing a serious error due to the above-mentioned concerns.
But I do not distinguish between subjective and objective in the fashion you describe-not at all.
Measuring is subset of, a type of assessing. Assessments can be objective or subjective depending upon the type of assessing being done and with what instrumentation. Scientific or engineering with quite precise measuring instruments that take out the “human element” are as objective as it can be, but even then there are margins of error. But those margins are so small as to be insignificant. Hence for all practical purposes “objective”. But that does not occur in educational assessment.
Any assessment of human behavior (in the broadest sense of the word, including mental “behavior”) and/or assessments dealing with aesthetics cannot be measured in any fashion due to the fact mentioned above of no standard unit of measure. All educational assessment is subjective. There is no getting around that fact.
The other point is “What is the purpose of assessing in the teaching and learning process?” For too long now the dominant meme has been to use assessment as a diagnostic tool to determine abnormalities in learning. Abnormalities meaning any learning that deviates from the standard that is given against which the question is developed. That deviation can go in myriad directions. Why this “diagnostic” function and not another basic reason? That’s a question for another day.
For me, the only reason to have any assessment of the what a student learns is so that the student may learn more about his/her own learning, where they are at in learning what he/she wants and needs to learn. Using information/data from what a student is learning for any other purpose is wrong and unethical (and further discussion of that is for another day) as far as I am concerned, except for true diagnostic purposes in determining educational disabilities. Other than that all other assessments should only be to help the student better understand themselves.
LikeLike
The margins of error don’t make measurements subjective.
They only quantify the uncertainty.
Now, if someone makes decisions based on such measurements (eg, a policeman gives a ticket when his radar detector says a person is speeding) such decisions are necessarily subjective to a degree, but the measurements themselves are not, provided the results were not purposefully biased by the experimenter.
The more uncertainty there is associated with a measurement, the more subjective the decisions based on such measurements become.
Sometimes the uncertainty associated with a measurement is larger than the measured value itself, in which case one can not say that the measured value differs from zero.
When people ignore the uncertainty and make decisions as if there were no uncertainty, they kid themselves into believing that they are making “purely objective” decisions.
LikeLike
Thanks, that’s an excellent explanation about the margin of error although in the standards and testing regime I would say that “attempts” are made at supposedly quantifying that margin of error. Psychometricians would like us to believe that these attempts all have statistical validity. And perhaps they do, I’m not a statistician so I can’t verify one way or the other. But then to take that supposed statistical validity and start making statements about the test, test taker and/or the teacher as in VAM is one big cluster. . . .!
And you are correct that margins of error don’t make the measurement subjective. I did not mean to imply that and don’t think I did but your thought makes an important point.
And the counterpoint is “Why attempt to eliminate supposed error when it will always be there? Why not acknowledge whatever errors may occur in an assessing process (how often are magins of error ever given to the test taker in giving the result?-ha ha, eh), use that information to further the assessing process (which should be focused exclusively in helping the learner learn-see my comment above) and move on?
LikeLike
And to bring back one of your good ones, SDP:
Whatever is measured counts
Whatever counts is measured
And counting whatever measures
Is measuring whatever counts
LikeLike
If I actually said that, I must have lost count of the beers.
LikeLike
Argue with his statement: The closest we can come to measurement in testing is to count responses we judge correct to a series of answerable questions or count the number of completed tasks.
If this is true, the argument must turn to a comparison of the worth of different questions. At this point the subjectivity overcomes any objective facade we are holding up, and the crashing of our logical system is deafening. Is an essay worth a thousand multiple choice questions? A picture worth a thousand words? An invention better than a painting?
The older I get, the more I throw up my hands, despairing of ever being fair to anyone I am forced to evaluate as a student. Rubrics provide us with useless bases for comparison of tasks, filling us with false hubris.
The only evaluation we should ever attach to a child is the general opinion about what we personally feel they would succeed at. They could then make their own decisions.
LikeLiked by 1 person
“. . . filling us with false hubris.”
Exactly. Great line!
What you have stated really does point to some of the many problems, errors as Wilson calls them in the whole standards and testing charade, or educational malpractices as I and others call it.
Keep the comments coming and we’ll all keep reading and enjoying, and for some, hopefully, learning much needed thoughts/lessons.
LikeLike
Duane
I think a large part of the confusion between measurement and assessment is caused by the sloppy use of terminology.
As you have pointed out many times, what people refer to as “measurements” of educational achievement or growth are actually assessments.
The very word “assess” presumes an “assessor” to do the assessing. That means there is always subjectivity involved.
As mathematician Cathy O’Neil (aka Mathbabe) makes clear in her book, Weapons of Math Destruction, even the assessments performed by computers are subjective because the algorithms used to do the assessing were designed by humans and are subjective.
I think it is actually very unfortunate that measurement is used to indicate either an actual scientific measurement (speed with a radar instrument) or what is actually an assessment.
If “measurement” were reserved strictly for those things that can actually be measured with instrumentation (speed, weight, mass, charge, volume, area, etc) and assessment for those things that involve a human “assessor” (either directly or indirectly in the case of the algorithms), there would be much less confusion and people would be much less likely to take stuff like IQ and other standardized test scores seriously because it would be clear that they are not real measurements and are subjective by their very nature.
But I know it’s too late to restrict the definition of measurement at this point.
LikeLiked by 1 person
The basic distinction that I would draw between measurement and assessment is that measurement does not involve decision, but assessment does.
As I indicated above with the example of the policeman checking for speeding, measurements can be used to make decisions, which means they can be used for assessments, but they are not actually assessments.
LikeLike
Excellent points. You’ve said in quite far fewer words what I’ve been getting at for years. Thanks! I discuss the problem of the usage of both the terms standard and measure in Ch. 6 of my book. If you would like a copy of it please feel free to contact me at duaneswacker@gmail.com and we can arrange to get you a copy.
However to give up the battle on language usage is to lose the war on the war on public education. I contend that until we can win the battle of “meaning” we will continue the Quixotic/Sysyphean task of battling without success the chimeras that are the educational malpractices of standards and testing.
LikeLike
Thanks to you Duane for making me think about this stuff.
The whole subject of measurement is fascinating and is at the core of science.
But even some folks who should know better have not really thought much about what it means to be an actual measurement.
For example, you might read claims that measurements (eg, of spiral galactic rotations) have been performed that show the presence of ” dark matter” ( which, as the name indicates can not be detected by normal means — eg, through optical observation.
But these measurements actually do no such thing.
What the measurements actually show is that there is something missing — either in what we can detect thru direct non-gravitational means OR there is something wrong with the currently accepted theory to describe gravitational interactions (Einstein’s general theory of relativity).
As of yet, no direct measurement has shown the existence of dark matter.
Any conclusions are based on indirect (gravitational) effects that can not be explained with General relativity (and Newton’s laws, which are subsumed by GR).
So, the “measurement” of dark matter is not really a measurement at all. It’s an assessment based on measurements of galactic rotations and assumptions about how gravity works.
LikeLike
Onto-epistemological questions and concerns abound, eh!
LikeLike
Assessing…
I just know what he thinks.
I just know what be believes.
I just know what he meant.
I just know what he needs, to believe, what he thinks, I meant.
Then I had dreams that made no sense.
What did they mean?
Did they reflect what I believe?
What was I thinking?
Was I thinking?
I just know…
I’ve been through the psychometric- fudge- categorisation mill.
I’ve passed the tests.
I’ve met the standards.
I’ve been assessed.
Now I await the return of the favor…
LikeLike
From Wilson:
“The examination is the ceremony of ordering; it is the mechanism through which real people (and hence the world) is ordered, and held in order, in all of the meanings of that word. By doing this in a setting in which the person who establishes order is also the person who establishes truth through knowledge, the certainty of correctness is established, and the person becomes an object in the acceptance of their place in the line, in their acceptance of their uni-dimensionality, in their incorporation of their relative merit as an essential part of their beingness.
Of course the examination is also a crucial element in the construction of human cognition. It defines what are true and false facts, what is right and wrong thinking, and what are the acceptable limits of intuition and feeling. But we are more concerned here with social categorisation.
The report is the place where such individuality is made official; here is the permanent record, uncorrupted by any possibility of error, of one’s place in the order of things; of a person’s history, present, and future distilled into a single mark; of a sign that evokes possibilities and defines exclusions; in the world of higher education and the world of work, here is the official indicator of who you are, what you are.
Foucault indicates that this individualisation through comparison is
intensified as power disperses and abnormality increases:
‘as power becomes more anonymous and more functional, those on
whom it is exercised tend to be more strongly individualized; it is
exercised by surveillance rather than ceremonies, by observation
rather than commemorative accounts, by comparative measures that
have the ‘norm’ as reference rather than genealogies giving ancestors
as points of reference; by ‘gaps’ rather than by deeds. In a system of
discipline, the child is more individualized than the adult, the
patient more than the healthy man, the madman and the delinquent
more than the normal and non-delinquent (p193).’
It is at these crucial points that define exclusion that any error becomes unacceptable. These are the points that define, not so much the norm, but the gaps that define abnormality, unacceptability, dangerous deviance. The normal is indeed defined by a broad grey band, but it is essential that the abnormal be determined by the thin red line that separates. And that line, that thin red line where the blood flows, is the standard.
LikeLike
A few thoughts –
Leftover from the other day (sorry) – – there are teachers who should not be teaching. They should be provided substantive support and guidance. If they are still “ineffective” (not attempting to define that here) they shouldn’t be teaching. For the critics – please write in and state that you would gladly put your daughter, son, niece, nephew… in ANY classroom because every teacher is “good.”
However – using test scores – VAMed, stanined, quadranted, above proficiencied, or any other manipulation of what are already unreliable results is NOT the way to scrutinize teachers.
Suggestions:
1. As a component of the evaluation process (component – one aspect – not a percentage) teachers should have to bring in evidence their kids are learning. Essays. Projects. Videos of science experiments. Kids reading or speaking or working in groups. Maybe a teacher made unit test.
VAM is so-called science. Our so-called president DOES NOT BELIEVE IN SCIENCE. He is also incompetent (“fire that employee in the situation room”), uninformed, refuses briefings unless there are flash cards, and loves to go against the mainstream.
And, Mr. Duncan was appointed by President Obama so if Arne likes something, this president hates it without understanding it.
So – someone ask the president “Why do you allow this junk science perpetuated by Obama’s appointee to let Washington dictate how to evaluate our red, white, and blue small town teachers?”
Executive order. VAM’s gone!
LikeLike
Leverage their idiocy to achieve our own goals.
I like it!
LikeLike
With that kind of thinking you might convince me that you’re not an adminimal, Wait? What?.
LikeLike
It’s interesting how education is a factor in the Wisconsin governor’s race. I can’t figure out why anyone would vote for Walker.
…………………….
The Wisconsin Governor’s Race Might Be Decided by Education..The Atlantic
Voters are “sick and tired of people playing political games with kids in schools.”
Voters think about a lot of things at the polls: immigration, the economy, health care, gun policy, and—more cynically—party affiliation. But education is an issue that doesn’t typically poll near the top of the list, even though it’s often thought of as a bedrock of society. Tony Evers, Wisconsin’s public-schools chief, and the most likely Democratic candidate to take on Governor Scott Walker in November’s gubernatorial election, is banking on the fact that that’s changing.
Evers, 67, isn’t the only educator who says he’s been inspired to seek political office because of the status quo—underfunded schools that have been hit with spending cuts, low teacher pay, and perpetual achievement gaps…
For his part, Evers has had virtually every school-related job you can imagine: He’s been a teacher, principal, and superintendent over his more than three decades in education. Running for governor was the farthest thing from his mind just a short few years ago, he says. But that changed after he won a third term as the state’s superintendent of public instruction. “It was clear to me that as much as I love my job as state superintendent—I think it’s a very important one—there are things that I just cannot accomplish for the kids of this state in my present role,” he told me. “And I don’t believe that Scott Walker will deliver on any promise he has around education.”…
Read More:
http://www.theatlantic.com/education/archive/2018/08/wisconsin-governors-race-tony-evers-scott-walker-education/566600/?utm_source=eb
LikeLike