Archives for category: Research

 

Leonie Haimson is a fearless advocate for students, parents, and public schools. She runs a small but mighty organization called Class Size Matters (I am one of its six board members), she led the fight for student privacy that killed inBloom (the Gates’ data mining agency), and she is a board member of the Network for Public Education. None of these are paid positions. Passion beats profits.

 

In this post on the New York City parent blog, she takes a close look at a new report that lauds the Bloomberg policy of closing public schools as a “reform” strategy. The report was prepared by the Research Alliance at New York University, which was launched with the full cooperation of the by the New York City Department of Education during the Bloomberg years (Joel Klein was a member of its board when it started).

 

Haimson takes strong exception to the report’s central finding–that closing schools is good for students–and she cites a study conducted by the New School for Social Research that reached a different conclusion. (All links are in the post.)

 

Furthermore, she follows the money–who paid for the study: Gates and Ford, then Carnegie. Gates, of course, put many millions into the small schools strategy, and Carnegie employs the leader of the small schools strategy.

 

Haimson writes:

 

“The Research Alliance was founded with $3 million in Gates Foundation funds and is maintained with Carnegie Corporation funding, which help pay for this report. These two foundations promoted and helped subsidize the closing of large schools and their replacement with small schools; although the Gates Foundation has now renounced the efficacy of this policy. Michele Cahill, for many years the Vice President of the Carnegie Corporation, led this effort when she worked at DOE.

 

“The Research Alliance has also been staffed with an abundance of former DOE employees from the Bloomberg era. In the acknowledgements, the author of this new study, Jim Kemple, effusively thanks one such individual, Saskia Levy Thompson:

 

[He wrote:] ‘The author is especially grateful for the innumerable discussions with Saskia Levy Thompson about the broader context of high school reform in New York City over the past decade. Saskia’s extraordinary insights were drawn from her more than 15 years of work with the City’s schools as a practitioner at the Urban Assembly, a Research Fellow at MDRC, a Deputy Chancellor at the Department of Education and Deputy Director for the Research Alliance.’

 

Levy Thompson was Executive Director of the Urban Assembly, which supplied many of the small schools that replaced the large schools, leading to better outcomes according to this report — though one of these schools, the Urban Assembly for Civic Engagement, is now on the Renewal list.

 

After she left Urban Assembly, Levy Thompson joined MDRC as a “Research Fellow,” despite the fact that her LinkedIn profile indicates no relevant academic background or research skills. At MRDC, she “helped lead a study on the effectiveness of NYC’s small high schools,” confirming the efficacy of some of the very schools that she helped start. Here is the first of the controversial MRDC studies she co-authored in 2010, funded by the Gates Foundation, that unsurprisingly found improved outcomes at the small schools. Here is my critique of the follow-up MRDC report.

 

“In 2010, Levy Thompson left MRDC to head the DOE Portfolio Planning office, tasked with creating more small schools and finding space for them within existing buildings, which required that the large schools contract or better yet, close.

 

“And where is she now? Starting Oct. 5, Saskia Levy Thompson now runs the Carnegie Corporation’s Program for “New Designs for Schools and Systems,” under LaVerne Evans Srinivasan, another former DOE Deputy Chancellor from the Bloomberg era Here is the press release from Carnegie’s President, Vartan Gregorian:

 

“‘We are delighted that Saskia, who has played an important role in reforming America’s largest school system, is now joining the outstanding leader of Carnegie Corporation’s Education Program, LaVerne Evans Srinivasan, in overseeing our many investments in U.S. urban education.'”

 

Concludes Haimson:

 

“How cozy! In this way, a revolving door ensures that the very same DOE officials who helped close these schools continue to control the narrative, enabling them to fund — and even staff — the organizations that produce the reports that retroactively justify and help them perpetuate their policies.”

 

 

The American Educational Research Association issued a warning against the use of value added measures for high-stakes decisions regarding educators and teacher preparation programs. The cardinal rule of assessment is that tests should be used only for the purpose for which they were created. A measure of fourth grade reading measures the student, not the teacher, the principal, or the school.

 

 

AERA Issues Statement on the Use of Value-Added Models in Evaluation of Educators and Educator Preparation Programs
WASHINGTON, D.C., November 11—In a statement released today, the American Educational Research Association (AERA) advises those using or considering use of value-added models (VAM) about the scientific and technical limitations of these measures for evaluating educators and programs that prepare teachers. The statement, approved by AERA Council, cautions against the use of VAM for high-stakes decisions regarding educators.

 

In recent years, many states and districts have attempted to use VAM to determine the contributions of educators, or the programs in which they were trained, to student learning outcomes, as captured by standardized student tests. The AERA statement speaks to the formidable statistical and methodological issues involved in isolating either the effects of educators or teacher preparation programs from a complex set of factors that shape student performance.

 

“This statement draws on the leading testing, statistical, and methodological expertise in the field of education research and related sciences, and on the highest standards that guide education research and its applications in policy and practice,” said AERA Executive Director Felice J. Levine.

 

The statement addresses the challenges facing the validity of inferences from VAM, as well as specifies eight technical requirements that must be met for the use of VAM to be accurate, reliable, and valid. It cautions that these requirements cannot be met in most evaluative contexts.

 

The statement notes that, while VAM may be superior to some other models of measuring teacher impacts on student learning outcomes, “it does not mean that they are ready for use in educator or program evaluation. There are potentially serious negative consequences in the context of evaluation that can result from the use of VAM based on incomplete or flawed data, as well as from the misinterpretation or misuse of the VAM results.”

 

The statement also notes that there are promising alternatives to VAM currently in use in the United States that merit attention, including the use of teacher observation data and peer assistance and review models that provide formative and summative assessments of teaching and honor teachers’ due process rights.

 

The statement concludes: “The value of high-quality, research-based evidence cannot be over-emphasized. Ultimately, only rigorously supported inferences about the quality and effectiveness of teachers, educational leaders, and preparation programs can contribute to improved student learning.” Thus, the statement also calls for substantial investment in research on VAM and on alternative methods and models of educator and educator preparation program evaluation.

 

Related AERA Resource:
Special Issue of Educational Researcher (March 2015)—
Value Added Meets the Schools: The Effects of Using Test-Based Teacher Evaluation on the Work of Teachers and Leaders

Kenneth Zeichner and Hilary G. Conklin complain that vendors of alternative pathways into teaching have been misusing research to slam university-based teacher education. In an excerpt from a longer study, they document how organizations like Teach for America, the National Council on Teacher Quality, and the Relay “Graduate School of Education” have selectively quoted research to support their own self-interest. They seek not to improve university-based teacher education, but to replace it with entrepreneurial programs.

Zeichner is a professor of teacher education at the University of Washington, Seattle, and professor emeritus in the School of Education at the University of Wisconsin-Madison. A member of the National Academy of Education, he has done extensive research and teaching and teacher education. Conklin is a program leader and associate professor of secondary social studies at DePaul University whose research interests include teacher learning and the pedagogy of teacher education.

They write:

Critics of college and university-based teacher preparation have made many damaging claims about the programs that prepare most U.S. teachers–branding these programs as an “industry of mediocrity”–while touting the new privately-financed and- run entrepreneurial programs that are designed to replace them. These critics have constructed a narrative of failure about college and university Ed schools and a narrative of success about the entrepreneurial programs, in many cases using research evidence to support their claims.

Yet in a recent independently peer-reviewed study that will be published in Teachers College Record, we show how research has been misused in debates about the future of teacher education in the United States. Critics have labeled university teacher education programs failures and decreed their replacements successes by selectively citing research to support a particular point of view (knowledge ventriloquism), and by repeating claims based on non-existent or unvetted research, or repeatedly citing a small or unrepresentative sample of research (echo chambers).

After citing specific examples of the misuse of research, they make the following recommendations:

In order to hold all programs — public and private — to common standards of quality and evidence, we believe that several things need to be done to minimize the misuse of educational research.

First, all researchers who conduct studies that purport to offer information on the efficacy of different program models, and those who produce syntheses of studies done by others, should reveal their sources of funding, their direct and indirect links to the programs, and they should subject their work to independent and blind peer review.

Second, given that much academic research on education is inaccessible to policymakers, practitioners, and the general public, researchers should take more responsibility for communicating their findings in clear ways to various stakeholders.

Third, the media should cover claims about issues in teacher education in proportion to the strength of the evidence that stands behind them and whether or not they are supported by research that has been independently vetted.

Fourth, we should assess the quality of programs based on an analysis of a variety of costs and benefits associated with particular programs, and not just look at whose graduates can raise test scores the most. Research suggests that an emphasis only on raising test scores deepens educational inequities and continues to create a second-class system of schooling for students living in poverty.

Terry Marselle has written a research-based review of the Common Core standards. The title is: “Why the Common Core is Psychologically and Cognitively Unsound.” His book is available on amazon for FREE on Kindle from June 1-5.

Linda McNeil of Rice University has started a new blog that you should read regularly. Linda is a respected scholar of high-stakes testing and its inequitable effects. Read this paper, for example, “Avoidable Losses,” which she co-authored.

She writes:

“I want to personally invite you to visit my new blog, Educating All Our Children. This project is my response to the need I perceive for a place to bring people together around the issues of public schooling, equity and high-stakes testing. I am reaching out to you, a group that feels as passionate about equality in our schools as I do. Many of you have already shown me much support in conceptualizing this blog, for which I am very grateful.

“I have a tag line for my blog: Bringing together research, analysis, advocacy and community on behalf of the public’s schools. I’m writing in part to ask you to help me with this goal by sending me important links, leads, new stories and information that I can use to advance the movement. I would also ask that you forward, share and circulate my posts with people who share our interests. I’m not on Facebook (yet), but you can add me on Blogger and Google+ and subscribe by email to my blog.

“Thanks again for all your support over the years, and I look forward to your participation in my new venture. I hope you will visit regularly and comment often.”

Here is a sample of one of her recent posts.

Grover “Russ” Whitehurst was ousted from his job as head of the Brown Center on Education Policy at the Brookings Institution. Whitehurst had previously been research director of the Institute of Education Sciences in the George W. Bush administration. He was, of course, a big supporter of testing and choice. He even created an annual ranking of the districts with the most school choice. He gave Brookings, once known as a liberal think tank, a rightwing gloss.

In 2012, Whitehurst fired me as a Senior Fellow at Brookings, an unpaid position. I had been at Brookings since 1993. From 1993-95, I was in residence and wrote a book there on national standards. I had originally been offered the Brown Chair at Brookings but turned it down because I wanted to return to Néw York City.

Whitehurst said he was removing me from my unpaid position because I was “inactive.” At the time, my book “Death and Life of the Great American School System” was ranked as the #1 social policy book on amazon. Inactive? Hardly.

The day I was terminated, I posted on the Néw York Review of Books blog a piece that was highly critical of Mitt Romney. Whitehurst was a Romney adviser. A few hours later, I received an email from Whitehurst saying I was no longer a Senior Fellow due to my inactivity. He later said there was no connection between my anti-Romney post and his decision to drop me from a post I had held for many years, without any notice or conversation.

It will be interesting to see who replaces Whitehurst, whether liberal, conservative, neoliberal, neoconservative.

Matthew Di Carlo of the Albert Shanker Institute of the American Federation of Teachers is neither pro-TFA nor anti-TFA.

 

Here he reviews the latest study of TFA by Mathematica.

 

It has been widely reported that the study found little or no difference between the test scores of the students taught by TFA and by regular teachers. TFA saw that as a victory, since it presumably showed that no training or experience was needed to achieve the same results. Others saw it as a repudiation of TFA’s oft-repeated claims that their recruits were superior to career teachers.

 

Di Carlo parses the results and reaches this conclusion:

 

Now, on the one hand, it’s absolutely fair to use the results of this and previous TFA evaluations to suggest that we may have something to learn from TFA training and recruitment (e.g., Dobbie 2011). Like all new teachers, TFA recruits struggle at first, but they do seem to perform as well as or better than other teachers, many of whom have had considerably more experience and formal training.

 

On the other hand, as I’ve discussed before, there is also, perhaps, an implication here regarding the “type” of person we are trying to recruit into teaching. Consider that TFA recruits are the very epitome of the hard-charging, high-achieving young folks that many advocates are desperate to attract to the profession. To be clear, it is a great thing any time talented, ambitous, service-oriented young people choose teaching, and I personally think TFA deserves credit for bringing them in. Yet, no matter how you cut it, they are, at best, only modestly more effective (in raising math and reading test scores) than non-TFA teachers.

 

This reflects the fact that identifying good teachers based on pre-service characteristics is extraordinarily difficult, and the best teachers are very often not those who attended the most selective colleges or scored highly on their SATs. And yet so much of our education reform debate is about overhauling long-standing human resource policies largely to attract these high-flying young people. It follows, then, that perhaps we should be very careful not to fixate too much on an unsupported idea of the “type” of person we want to attract and what they are looking for, and instead pay a little more attention to investigating alternative observable characteristics that may prove more useful, and identifying employment conditions and work environments that maximize retention of effective teachers who are already in the classroom.

 

For me, the problem with all such studies is the assumption that the best (perhaps the only) way to identify the best teachers is by comparing changes in test scores. Great teachers supposedly get higher scores than mediocre teachers. I think that places far too much faith in standardized testing and in the assumption that education is solely measured by those tests. It makes the tests the arbiters of all things, even though most teachers do not teach tested subjects. Test-based findings are even more suspect when the children are very young.

Julian Vasquez Heilig, who recently moved from the University of Texas to California State University at Sacramento, is one of the nation’s leading authorities on Teach for America. He has studied their performance over time (see here and here), and he is not a fan. When Mathematica released its latest study of TFA, Heilig read it closely and analyzed the findings. TFA boasted that the study showed that its teachers were just as good as those who had studied education and intended to be career teachers. Some readers gleaned from this finding that “anyone can teach, no professional preparation needed,” that is, if they graduate from a highly selective college and are admitted to TFA.

 

Heilig digs deeper and has a different take on the study. The main finding, he says, is that Mathematica found no statistically significant differences in the groups of teachers they studied. However, he points out, the TFA teachers were overwhelmingly white, and few had any intention of staying in teaching as a career.

 

He notes that the test of “effectiveness” in pre-K-grade 2 is a five minute test:

 

Equally effective at what?…Mathematica utilized performance on the Woodcock Johnson III for the Pre-K-2 results— which takes 5 minutes to administer. Thus, the effectiveness of TFA teachers compared to Pre-K – 2nd grade teachers is based on a five minute administration to capture letter-word identification (Pre-K – 2) and applied problems for mathematics Pre-K – 2). Furthermore, one of the more egregious issues in the study is the aggregation of grades is that of the states that have Pre-K programs, more than half of states do not even require Pre-K teachers to have a bachelor’s degree. The report does not state that lack of a degree was an exclusion criteria and it is explicit that community preschools were included, so it appears than an aggregate that includes not only alternatively certified but also non-degreed teachers worked to TFA’s advantage. Should we really be impressed that TFA teachers outperformed a group that could have included non-degreed teachers? And they do it twice: with kindergarten and with grades K, 1, and 2.

 

What are the lessons of the study? Heilig writes:

 

So the [TFA] teachers were— on average— young, White, and from selective colleges. They had not studied early childhood in college and had very little teaching experience. They reported a similar amount of “pedagogy” (primarily the 60 hours from the five week Summer Institute), and more professional development (as we discussed above, they viewed it not very valuable). TFA teachers also reported less student teaching experience before they entered the classroom. They also were more likely to be working with a formal mentor (I mentioned David Greene’s point about the drain on mentors due to the constant carousel of Teach For America teachers in and out of schools here). As new teachers, they spent more time planning their own lessons, but were less likely to to help other teachers. Finally, TFA teachers were less satisfied “with many aspects of teaching” and less likely to “plan to spend the rest of the career as a classroom teacher….”

 

In conclusion, read at face value, here is the message Mathematica appears to promulgate with the report:

 

We do not need experienced (read: more expensive) teachers when non-experienced, less expensive teachers get the “same” —though not statistically significant— outcomes.
We do not need a more diverse workforce of teachers, again, because TFA teachers, who are overwhelmingly white, get the same outcomes.
Is TFA really in alignment with a vision for providing every student a high quality teacher? Or do they, Mathematica et al. just keep telling us that they are?

 

For myself, I have read many times that Teach for America invites young people to “make history” by serving for two years. And Wendy Kopp has frequently said that “One day,” all children in America will have an excellent teacher. I have a hard time understanding the logic of these claims. If the TFA teachers get the same results as current teachers, how is that “making history”? If most TFA recruits leave after two years, how does that lead to the conclusion that one day all children will have an excellent teacher? If TFA persuades policymakers that teachers can do a good enough job with no professional preparation, doesn’t that decimate the idea of teaching as a profession? If anyone can teach so long as they went to a selective college, how does that raise the standard for teachers? If our policymakers prefer churn, with teachers leaving every two or three years to find their real career, how is that good for students? How does TFA improve the profession? It doesn’t. It eliminates it.

 

For his fearlessness, for his willingness to stand up to those with money and power, for his willingness to present the evidence as he finds it without fear or favor, I place Julian Vasquez Heilig on the honor roll of this blog. He is an example to all researchers of the ethics of his profession. To be an outstanding researcher requires years of study, scholarship, discipline, dedication, and experience. Sort of like being a great teacher.

 

 

 

 

A new study by Mathematica Policy Research finds that young corps members in Teach for America get no better results than other teachers.

 

Normally, this would not be big news, since TFA teachers have only five weeks of training. But for years, TFA has boasted that their young people were far superior to other teachers who had gone through professional preparation programs. Now, TFA leaders are claiming to be satisfied that their five weeks of training allows them to do just as well as those who spent a year or more learning to teach. The implicit logic of their perspective is that teaching is not a profession and that no preparation is needed beyond five weeks of TFA training. However you slice it, the TFA message degrades the profession. No profession would be considered to be a profession if any bright young person could succeed with only a few weeks of preparation. One cannot even imagine doctors or lawyers or accountants boasting that they were successful with a five-week training program.

 

The Mathematica study may not end the debate about the value of TFA. Its biggest fans seem to be the Walton Foundation, the Broad Foundation, and other foundations that want to support the proliferation of non-union charter schools with low costs and high teacher turnover. Walton gave $50 million to TFA; Broad collected $100 million from a group of foundations for TFA. And Arne Duncan gave TFA $50 million. TFA’s special contributions to American education, it appears, are to staff non-union charter schools and to demonstrate that teaching is not a profession.

The New York State Education Department released educator evaluation results on February 26, and once again, the overwhelming majority of teachers received effective or highly effective ratings. State officials were deeply disappointed by the overwhelmingly positive results. They seem to operate under the assumption that poor test results must be caused by “bad” teachers, and that their evaluation program should identify them so they may be fired.

 

The SED found that:

 

The final evaluation results show more than 95 percent of teachers statewide are rated effective (54 percent) or highly effective (42 percent); 4 percent are rated as developing; 1 percent are rated ineffective. Ninety-four percent of principals are rated effective (66 percent) or highly effective (28 percent).

 

The results were somewhat different in New York City, which used a plan imposed by then-State Commissioner John King:

 

New York City, whose evaluation plan was imposed by former Commissioner King when the New York City Department of Education could not reach agreement on the terms of the evaluation plan with the teachers union, showed greater differentiation than most districts in the State. Although New York City teachers and principals were evaluated on the same overall subcomponents as the rest of the State, the three subcomponents used different scoring ranges to determine the subcomponent rating categories (i.e., Highly Effective, Effective, Developing, Ineffective). Less than 10 percent of teachers in the city are rated Highly Effective, while 83 percent are rated Effective, 7 percent are Developing and 1 percent are Ineffective.

 

The leader of the state Board of Regents expressed disappointment at the high proportion of teachers found to be effective or highly effective:

 

“The ratings show there’s much more work to do to strengthen the evaluation system,” Board of Regents Chancellor Merryl H. Tisch said. “There’s a real contrast between how our students are performing and how their teachers and principals are evaluated. The goal of the APPR process is to identify exceptional teachers who can serve as mentors and role models, and identify struggling teachers to make sure they get the help they need to improve. The evaluation system we have now doesn’t do that. The ratings from districts don’t reflect the struggles our students face to achieve college and career readiness. State law must be changed to build an evaluation system that supports teaching and learning in classrooms across the State. Our students deserve no less.”

 

Chancellor Tisch, like Governor Cuomo, assumes that the proportion of students getting low scores should somehow be matched by a similar proportion of low-rated teachers. It would be useful if Chancellor Tisch and Governor Cuomo reviewed two basic documents: the American Statistical Association statement on the uses and misuses of value-added measurement (VAM) and the joint statement of the National Academy of Education and the American Educational Research Association. It is unfortunate that the Board of Regents and the Governor proceed without regard to research on the effects of out-of-school and in-school factors that affect test scores. Were they to familiarize themselves with the two documents cited, they might develop a very different action plan, one that helps both students and teachers.

 

See state ratings here.