In copying the response of Hart Research, I inadvertently copied only part of Guy Moyneaux’s comments.
Here is his full response:
TO:American Federation of Teachers
FROM:Guy Molyneux, Hart Research Associates
DATE:May 10, 2013
RE:Methodology for Common Core Survey
Following are some facts about the methodology for AFT’s recent survey of AFT K-12 teachers on Common Core implementation that may help to answer the criticisms and questions raised by Mercedes Schneider.
Schneider’s objections speak to two distinct questions: 1) does the survey reflect the views of AFT K-12 teachers?, and 2) if so, can the AFT results be extrapolated to all U.S. teachers? The answer to the first question is “yes,” for reasons explained below. The answer to the second question is “not necessarily.” When Randi Weingarten refers to what “teachers” think about the Common Core, she is referring to AFT teachers. This shorthand is not meant to deceive anyone; if it were, the press release and various poll materials would not have stated so clearly and repeatedly that the survey was conducted only among AFT members. (Indeed, even the quote highlighted by Schneider mentions “a recent poll of AFT members.”)
In fact, it is likely that a survey of all U.S. teachers would report results broadly similar to what we found among AFT members, for reasons explained below. However, it is true that we cannot be sure of this unless further research is done among non-AFT teachers. Such research would be welcome.
• The survey employed a standard sampling methodology, used in countless surveys by many polling organizations. On behalf of AFT, Hart Research Associates conducted a telephone survey of 800 AFT K-12 teachers from March 27 to 30, 2013. Respondents were selected randomly from AFT membership lists. This process of random selection produces a representative sample, allowing us to generalize from the survey respondents to the larger population being sampled (in this case, all AFT teachers). There is nothing unusual or controversial about this method.
• A sample size of 800 teachers is appropriate and common. Schneider notes that “AFT/Hart only surveyed nine one-hundredths of a percent of the AFT membership (.09%),” and adds for emphasis: “Please don’t miss this. AFT did not survey even 10% of its membership before forming an opinion of teacher acceptance of CCSS.” In fact, a survey sample size of 800 is reasonable and quite common: for example, most national media surveys interview between 800 and 1,000 registered voters. Moreover, researchers understand that survey samples are not properly evaluated as a percentage of the underlying population. By randomly selecting respondents, a relatively small sample can provide an accurate measurement on a much larger population. If Schneider’s 10% standard were correct, pollsters would need to interview 20 million U.S. voters to conduct a single survey of registered voters. Needless to say, not many surveys would be conducted.
• A reported margin of error of +/-3.5 percentage points does not indicate a lack of precision or poorly written questions. Schneider asks “How is it that a research firm only handling 800 surveys cannot get a more precise reading of the data than this? [a +3.5% margin of error]” and notes that “error is introduced in a lack of either question quality or precision in answering format, or both.”
The margin of error reflects the possibility that any single survey sample will not be perfectly representative of the full population. In this case, there is a 95% chance that a survey of all AFT teachers would yield results within 3.5 percentage points of those found in this survey. Schneider is correct that this means that AFT teachers’ approval of the Common Core State Standards could be as low as 71% or as high as 79% (and a 5% chance the proportion is even higher or lower). The margin of error has nothing whatsoever to do with question wording.
• The survey sample is demographically similar to the population of AFT teachers. In terms of age, gender, school type, and other demographic factors, the survey respondents closely resemble the larger population of AFT teachers. This information is available to anyone upon request. Schneider guesses that 95% of respondents reside in New York State, and criticizes the failure to disclose this “fact.” In reality, 36% of survey respondents live in New York, reflecting the geographic distribution of AFT members. As it happens, approval of the CCSS is actually somewhat higher – 82% – among AFT teachers outside of New York.
• A demographic breakdown of the survey sample, and precise question wording for all questions, is available upon request. Schneider claims that “Weingarten presents the results of her survey in suspiciously general terms” and faults her failure to provide comprehensive demographic information “at the outset of the study.” These survey results were presented not in a refereed academic journal, but in a simple Powerpoint slide show designed for a lay audience. There is no obligation to burden readers with exhaustive methodological details there. What is required is disclosure of this information upon request. The AFT does that. Schneider could have received answers to many of her questions – and saved herself a lot of time – by sending an email.
• It is likely that non-AFT teachers have similar views as AFT members, but we can’t be sure. AFT teachers are not demographically representative of all U.S. teachers: for example, they are more likely than average to teach in urban school districts. And of course they are union members.
However, the survey reveals support for the CCSS that is generally similar across most relevant demographic categories. For example, within AFT, 76% of urban teachers and 73% of non-urban teachers approve of the CCSS. For that matter, 71% of urban teachers and 78% of non-urban teachers share the worry that they will be held accountable for results on new assessments before instructional practice is aligned with the new standards. In general, the outlook of urban and non-urban AFT teachers on these issues appears to be more similar than different. The same is true in terms of region of the country. So it is likely that a survey of non-AFT teachers would yield similar findings. However, we can’t know that for sure without further research.
1724 Connecticut Avenue, N.W., Washington, D.C. 20009 202-234-5570 http://www.hartresearch.com
