A few days go, Professor Ira Shor posted a comment and asked if Mercedes Schneider would analyze the poll showing that 75% of AFT teachers support the Common Core standards. Mercedes Schneider saw his request in the comments section and posted her analysis. Schneider is a high school teacher in Louisiana with a doctorate in statistics and research methods.
Hart Research Associates, which conducted the poll, did not agree with Schneider. The Hart firm is a highly respected polling organization. I invited them to respond to Scneider’s review of their work, and they agreed to do so.
Their response begins here:
TO: American Federation of Teachers
FROM: Guy Molyneux, Hart Research Associates DATE: May 10, 2013
RE: Methodology for Common Core Survey
Following are some facts about the methodology for AFT’s recent survey of AFT K-12 teachers on Common Core implementation that may help to answer the criticisms and questions raised by Mercedes Schneider.
Schneider’s objections speak to two distinct questions: 1) does the survey reflect the views of AFT K-12 teachers?, and 2) if so, can the AFT results be extrapolated to all U.S. teachers? The answer to the first question is “yes,” for reasons explained below. The answer to the second question is “not necessarily.” When Randi Weingarten refers to what “teachers” think about the Common Core, she is referring to AFT teachers. This shorthand is not meant to deceive anyone; if it were, the press release and various poll materials would not have stated so clearly and repeatedly that the survey was conducted only among AFT members. (Indeed, even the quote highlighted by Schneider mentions “a recent poll of AFT members.”)
In fact, it is likely that a survey of all U.S. teachers would report results broadly similar to what we found among AFT members, for reasons explained below. However, it is true that we cannot be sure of this unless further research is done among non-AFT teachers. Such research would be welcome.
The survey employed a standard sampling methodology, used in countless surveys by many polling organizations. On behalf of AFT, Hart Research Associates conducted a telephone survey of 800 AFT K-12 teachers from March 27 to 30, 2013. Respondents were selected randomly from AFT membership lists. This process of random selection produces a representative sample, allowing us to generalize from the survey respondents to the larger population being sampled (in this case, all AFT teachers). There is nothing unusual or controversial about this method.
A sample size of 800 teachers is appropriate and common. Schneider notes that “AFT/Hart only surveyed nine one-hundredths of a percent of the AFT membership (.09%),” and adds for emphasis: “Please don’t miss this. AFT did not survey even 10% of its membership before forming an opinion of teacher acceptance of CCSS.” In fact, a survey sample size of 800 is reasonable and quite common: for example, most national media surveys interview between 800 and 1,000 registered voters. Moreover, researchers understand that survey samples are not properly evaluated as a percentage of the underlying population. By randomly selecting respondents, a relatively small sample can provide an accurate measurement on a much larger population. If Schneider’s 10% standard were correct, pollsters would need to interview 20 million U.S. voters to conduct a single survey of registered voters. Needless to say, not many surveys would be conducted.
1724 Connecticut Avenue, N.W., Washington, D.C. 20009 202-234-5570 http://www.hartresearch.com
Diane Ravitch's blog 
“But that’s the way most polling is done!” does not address the concerns expressed by Mercedes Schneider. We are not investigating teachers favorite breakfast cereal.
2old2tch: exactly! As this and the thirteen other comments blow attest, M. Schneider is spot on in her analysis and doubts.
I am baffled more than upset at such a shoddy defense. It is transparently obvious that those who commissioned the study felt that “75%” sounded a lot more convincing than “600.” Professional integrity should be worth more—a lot more—than that.
And given the hefty resources of the AFT, a significantly broader [and transparently neutral] sampling of opinion could have—and should have—been obtained on such an important issue as teacher attitudes about the CCSS. IMHO, this is more a failure of union leadership than a breach of professional ethics.
Props again to KrazyMathLady for stepping forward and averting an [I am genuinely sorry to say, sad] attempt at mathematical obfuscation and intimidation.
🙂
Where did this AFT poll take place? It aurely didn’t take place here where I live. I voted in no such poll as a member!
Teachers, Hart polled 800 teachers belonging to AFT. AFT is not in all 50 states. AFT is concentrated in New York. AFT boasts a membership of 1.5 million, over half of whom are teachers. According to Hart’s numbers, 600 of those teachers “support” CCSS (whatever that means, and however the question was put to them).
600 teachers. That’s all.
Even if Hart had surveyed 1% of AFT teachers, AFT is not equivalent to “nationwide.” The “larger population” to which their results can generalize can only be AFT members.
In my post, I posed a number of limitations to Hart’s study. They are not all addressed here. How many calls were hang-up calls? Not addressed. Question format was a crude dichotomy (yes/no). Question quality remains vague. How about that margin of error? There’s more detail in my objections than there is in this letter from Hart, a letter that only repeats much of what was already available in the AFT survey findings that I critiqued.
And when a person of such stature as Weingarten says “75%” of teachers polled support CCSS, the public does not know that this means 600 AFT teachers. They hear “75%” and think that 75% of all teachers, period, support CCSS. In her post on this blog, Weingarten says she endorses CCSS and tells teachers there’s nothing they can do; they’re just stuck with it.
I am tired of being told what I’m stuck with.
Teachers, parents, and schools are bucking up against “being stuck with” standardized testing. Why not pull the rug from under CCSS, as well?
Who out there supports CCSS? I’d really like to know. And I am curious what it is about CCSS that you like. Do you even know what CCSS is? Maybe that 600 will show up here to leave a comment.
I think some of the confusion lies in the difference between the headline of the referring to most teachers and the body of the post referring to AFT teachers. The sample was clearly not meant to be a random sample of all teachers, and the polling firm acknowledged that.
The margin of error, of course, comes from the fact that it was a sampling of the underlying population. A different set of randomly chosen teachers would likely result in a slightly different point estimate of the view of AFT teachers about the core even with exactly the same questions.
There should not be “confusion” when the results of this poll are being used to promote what amounts to endorsement for a national curriculum. Even a random sample of AFT presents problems if AFT is concentrated in some states and not others.
As to the margin of error, I know what it is, This margin of error could have been smaller had the researchers use da more sophisticated level of measurement (i.e. Likert scale) for their questions, And the questions themselves, if perceived as vague by the respondents, pose measurement error problems. Finally, HART does not offer info regarding just how much “larger” measurement error is for “subgroups.”
Lots of missing details for a study conducted by a supposedly “reputable” firm.
Mercedes,
Do we even know the exact wording of the questions? Was that posted anywhere?
Linda, so much about this study is not posted, including exact questions. Yet these folks insist they are reputable.
If I withhold details about my research, I am no longer reputable.
And this posters comment on your blog sheds some light and makes the results seem even murkier:
The Powerpoint doesn’t tell us the real question – only the part that sounds best. From what I can pull together, the actual question was:
“Common Core State Standards are a set of academic standards in English language arts and math for students in grades K-12 that have been adopted in most states.
“Based on what you know about the Common Core State Standards* and the expectations they set for children, do you approve or disapprove of your state’s decision to adopt them?”
Now remember, this is done on the phone. On the phone, it’s likely that many interviewee’s did not understand that “Common Core” is capitalized and refers to one very specific set of new standards that are beginning to be implemented. Instead, based on my experience in surveys like this, a significant portion of respondents ACTUALLY answered:
“Most states have adopted the common standards. Based on what you know about these standards, do you approve or disapprove of your state’s decision to adopt them?” (That’s what the human mind does while listening to a question over the phone.)
I don’t know about the New York concentration. We had a very large AFT membership in New Orleans. They had a big building and a credit union. We also had a strong AFT in Atlanta in some schools and a strong GAE in others—-strong enough to dispose of Gov. Roy Barnes, who was anti-teacher after one term. This is saying something in right-to-work states where there is no requirement to join a union.
Again, exact numbers regarding demographics are missing from this study. That is poor research practice.
She and her billionaire buddies appear desperate at this point. Someone posted this on another thread…a few back. It is clear Coleman is impressed with himself…he comes in around 11:00 and it appears he picks and chooses what “research” he reports and admires. He is a crude, pompous man with no respect for teachers or our children. He created national standards tied to national tests and works for a testing company, so what would we expect from him. He also states that students who do well on the SAT’s must be getting good grades…that is not actually always true. I actually don’t give a sh__ what he thinks.
[audio src="http://brightcove.vo.llnwd.net/e1/uds/pd/102148458001/102148458001_1997709561001_121129-StandardizedTest-64k-itunes.mp3" /]
I think a bigger sample was needed and they needed to survey all teachers, not just AFT members to get a representative sample, or at least all unionized teachers including those in NEA, but not A+Pel because they don’t claim to be a union. I would have been more comfortable with a survey of at least 10,000 teachers.
The smaller the sample relative to the population, the higher the chance of capitalizing on idiosyncracies of the individuals in the sample and the less likely that the sample truly represents the population.
Plus let’s say that AFT is in all 45 states endorsing CCSS. Assuming that the same number of surveys were conducted per state, that would be 800/45 = 17 or 18 teachers surveyed per state. So 75% endorsing CCSS would be, on average, 13 teachers per state.
Now I ask you, how ridiculous would it sound for Weingarten to visit a state and say that AFT did a survey and an overwhelming 13 teachers support CCSS? Not so impressive, huh? But by substituting “75%” for the actual number, Weingarten sounds like she has some real ammo.
Are you saying that 800 is not a large enough sample size? (If so, what number would be large enough in your view?) Or are you saying that the sample should have been stratified rather than random?
800 is not large enough. And stratification would be better than not in this case. But what I would really like to see (and should not have to ask for) to better inform my advice regarding sampling is information provided by AFT regarding actual teacher membership per state.
We all know how polls work. Usually to make a point for the person or group who initiate them. Randi would have been well advised to let this argument go.
I was selected for this poll. The caller would not identify the organization that authorized and paid for it. They did identify themselves as Hart. I was ready for a “rephormer” type poll, but was surprised when the questions clearly wanted to know if I had received enough training to implement CC. About halfway through the poll, I ended the conversation. I told the pollster that I felt the poll was leading my answers (push poll?). I am against the CCSS for many of the reasons discussed on this blog. The poll did not allow me to make that clear. It would have been impossible for the poll to determine that from the questions that were asked. It never asked if I support the CCSS, only if I thought it would improve student success (whatever that means). I admit, I do not remember the exact questions. But I am certain if those questions were made public, it would be embarrassing. Many of the questions were preceded by lengthy statements to set the situation (preambles). I hope that information is also discovered. I plan to record polls from now on.
The best way to deal with pollsters is to inform them that you will answer their questions after you have received $25/question. They will then hang up!
Mercedes, many thanks again for your brilliant good sense, very readable expertise, and clear view on the politics of this utterly treacherous moment for public life as well as public education.
Thanks, Ira. Hart and Weingarten should be embarrassed at their less-than-stellar responses to my work. I asked for details. They do not deliver.
Marcus of AFT felt obliged to copy the Hart letter into the comments section of my blog.
I graded it. C minus.
Diane posted a correction to the Hart response. More detail, but still a problematic study. I added comments to Diane’s reposting.
I also raised Hart’s grade to C.
Ira, I added a brief explanation regarding margin of error and measurement error to my original post. I included a link for the margin of error formula for categorical responses:
http://www.ehow.com/how_5276026_compute-margin-error-easy-methods.html
It clearly shows that survey responses figure into the formula, and therefore, instrument quality matters.
Just wanted to let you know since I wrote the original post with your inquiry in mind.
Mercedes….love your blog….it reads like a suspense novel. Brilliant. In TOTAL agreement about AFT’s intent to use percentages to sway/misinform.
Thank you for your work.
Thank you. The actual numbers are telling. Percentages are cloudy.
But you get an A+. Thanks.
Thanks, Carrie. I tend not to trust researchers who use the defense that details are not important “here” and that if one wants details, one must ask. It is poor research practice to require readers to ask. In this day of the ease of adding a link for a reader to see the details, requiring people to ask seems sloppy at best and deceptive at worst.
I agree with Schneider. I always show where I got my data. That is for credibility. I tell people not to trust me or anyone until they check out the sources and see that it is exactly what I told them it is. If they want credibility give us the information such as where did you call. Was it evenly spaced in the areas of their teachers nationally? What were the questions or question? How did the answers come out by region? Then we might have a sense of confidence. The big money is always lurking behind the “Wizard of Oz’s screen.” We are only effective by transparency and honesty. That is as it should be.
Considering that the CTU is IFT/AFT, I can hardly believe that ANY CTU members support the Common Core! Any of you in Chicago polled? Comments, please! Thanks!
I do not care what you do with your data or how you transform it to make it look good for you but I can assure you with certainty that the children of the USA are being used for guinea pigs and the teachers have been thrown under a bus.
The DOE’s are in bed with the $Book Companies..Period…and Period..
The DOE’s are basing the entire curriculum on the worst -one-size fits all tests that have ever been created..
Not sure where the wind will take me in my next journey. I do know that I learn so much through information like this… My brows were furrowed when I read the 75%… I too wondered… Thank you!
Thanks for the explanation. Now I understand why I was never polled. But here’s where this whole poll story falls flat: AFT polls its members via email and with polls posted on its website often. Why not that kind of a poll for such an important issue? Why spend the money on this telephone poll which only reached a fraction of the membership? Something doesn’t smell right here, AFT leadership.