Archives for category: Research

How many times have you read in a report or in the newspaper that X method or Y school was able to produce an extra 40 days or extra weeks of learning in reading or math?

 

How do gains in test scores get converted into days or weeks or months?

 

The answer, according to Gary Rubinstein, is that they don’t. Or they shouldn’t. It is nonsense.

 

I recently read a Mathematica Policy Research report on the Teacher Incentive Fund (merit pay), which claimed that a 1% increase in test scores was equivalent to an additional three weeks of learning. See here (study snapshot) and here (executive summary) and here (full report).

 
Performance Bonuses for Educators Led to Small Improvements in
Student Achievement

 

Educators’ understanding of bonus program improved, but challenges remain

 

New findings from Mathematica Policy Research show that a federal program providing bonuses to educators based on their performance had a small, positive impact on student achievement. In the first report to describe the effects of pay-for-performance bonuses within the Teacher Incentive Fund (TIF) program on student achievement, researchers found that student scores on standardized reading tests rose by 1 percentile point—the equivalent of about three weeks of additional learning. The study also showed similarly positive, but statistically insignificant, improvements in math.

 

I asked Gary if it made sense to translate a one-point gain into three weeks of learning, and he replied:

 

Mathematica should stop using that ‘weeks of learning’ metric. They use a calculation that says that average teachers don’t teach very much so that they maybe get the kids to increase their scores from 24 percent passing (if they did not teach anything) to 34 percent in the entire year. So each ‘point’, by that logic, amounts to about a sixth of the year. A teacher with merit pay, then who gets that extra ‘point’ would be teaching 10% more in that year which is an extra three weeks. I wish they would just give the raw score which people could relate to, like there were 50 questions on the test and students of people without merit pay got 25 correct and students of people with merit pay got 26 correct. Then people would be able to put these numbers into perspective and realize that they are not a big deal.

 

 

 

 

Blogger Chaz’s School Daze explains why the NYU study on the “success” of closing large high schools and replacing them with small high schools is bogus.

He writes:

“This week, NYU released a study showing that students fared better with the closing of the many large comprehensive high schools and replaced by the Bloomberg small schools. The basis for the study’s conclusion was the increased graduation rate from the small schools when compared to the closed schools. However, the study is fatally flawed since the graduation rate is a bogus parameter and easily manipulated by the school Principal to allow students to graduate academically unprepared for college and career. Let’s look at how schools manipulate the graduation rate.”

I posted Leonie Haimson’s critique of this Gates-funded study, which relied on the views and insights of those in charge of designing and implementing the policy in the NYC Department of Education.

Chaz points out that the study ignored the pressure on teachers in the new small schools to pass students; the pressure on principals to raise graduation rates; and the widespread use of fraudulent “credit recovery” to hand diplomas to low-performing students.

The data on graduation rates are made meaningless by these corrupt practices. The researchers did not see fit to examine nefarious ways of graduating students who were unprepared for college or careers.

Chaz points out that educators in Atlanta went to jail and lost their licenses for changing grades. Why was there no investigation or prosecution of equally serious actions in Néw York City?

 

Leonie Haimson is a fearless advocate for students, parents, and public schools. She runs a small but mighty organization called Class Size Matters (I am one of its six board members), she led the fight for student privacy that killed inBloom (the Gates’ data mining agency), and she is a board member of the Network for Public Education. None of these are paid positions. Passion beats profits.

 

In this post on the New York City parent blog, she takes a close look at a new report that lauds the Bloomberg policy of closing public schools as a “reform” strategy. The report was prepared by the Research Alliance at New York University, which was launched with the full cooperation of the by the New York City Department of Education during the Bloomberg years (Joel Klein was a member of its board when it started).

 

Haimson takes strong exception to the report’s central finding–that closing schools is good for students–and she cites a study conducted by the New School for Social Research that reached a different conclusion. (All links are in the post.)

 

Furthermore, she follows the money–who paid for the study: Gates and Ford, then Carnegie. Gates, of course, put many millions into the small schools strategy, and Carnegie employs the leader of the small schools strategy.

 

Haimson writes:

 

“The Research Alliance was founded with $3 million in Gates Foundation funds and is maintained with Carnegie Corporation funding, which help pay for this report. These two foundations promoted and helped subsidize the closing of large schools and their replacement with small schools; although the Gates Foundation has now renounced the efficacy of this policy. Michele Cahill, for many years the Vice President of the Carnegie Corporation, led this effort when she worked at DOE.

 

“The Research Alliance has also been staffed with an abundance of former DOE employees from the Bloomberg era. In the acknowledgements, the author of this new study, Jim Kemple, effusively thanks one such individual, Saskia Levy Thompson:

 

[He wrote:] ‘The author is especially grateful for the innumerable discussions with Saskia Levy Thompson about the broader context of high school reform in New York City over the past decade. Saskia’s extraordinary insights were drawn from her more than 15 years of work with the City’s schools as a practitioner at the Urban Assembly, a Research Fellow at MDRC, a Deputy Chancellor at the Department of Education and Deputy Director for the Research Alliance.’

 

Levy Thompson was Executive Director of the Urban Assembly, which supplied many of the small schools that replaced the large schools, leading to better outcomes according to this report — though one of these schools, the Urban Assembly for Civic Engagement, is now on the Renewal list.

 

After she left Urban Assembly, Levy Thompson joined MDRC as a “Research Fellow,” despite the fact that her LinkedIn profile indicates no relevant academic background or research skills. At MRDC, she “helped lead a study on the effectiveness of NYC’s small high schools,” confirming the efficacy of some of the very schools that she helped start. Here is the first of the controversial MRDC studies she co-authored in 2010, funded by the Gates Foundation, that unsurprisingly found improved outcomes at the small schools. Here is my critique of the follow-up MRDC report.

 

“In 2010, Levy Thompson left MRDC to head the DOE Portfolio Planning office, tasked with creating more small schools and finding space for them within existing buildings, which required that the large schools contract or better yet, close.

 

“And where is she now? Starting Oct. 5, Saskia Levy Thompson now runs the Carnegie Corporation’s Program for “New Designs for Schools and Systems,” under LaVerne Evans Srinivasan, another former DOE Deputy Chancellor from the Bloomberg era Here is the press release from Carnegie’s President, Vartan Gregorian:

 

“‘We are delighted that Saskia, who has played an important role in reforming America’s largest school system, is now joining the outstanding leader of Carnegie Corporation’s Education Program, LaVerne Evans Srinivasan, in overseeing our many investments in U.S. urban education.'”

 

Concludes Haimson:

 

“How cozy! In this way, a revolving door ensures that the very same DOE officials who helped close these schools continue to control the narrative, enabling them to fund — and even staff — the organizations that produce the reports that retroactively justify and help them perpetuate their policies.”

 

 

The American Educational Research Association issued a warning against the use of value added measures for high-stakes decisions regarding educators and teacher preparation programs. The cardinal rule of assessment is that tests should be used only for the purpose for which they were created. A measure of fourth grade reading measures the student, not the teacher, the principal, or the school.

 

 

AERA Issues Statement on the Use of Value-Added Models in Evaluation of Educators and Educator Preparation Programs
WASHINGTON, D.C., November 11—In a statement released today, the American Educational Research Association (AERA) advises those using or considering use of value-added models (VAM) about the scientific and technical limitations of these measures for evaluating educators and programs that prepare teachers. The statement, approved by AERA Council, cautions against the use of VAM for high-stakes decisions regarding educators.

 

In recent years, many states and districts have attempted to use VAM to determine the contributions of educators, or the programs in which they were trained, to student learning outcomes, as captured by standardized student tests. The AERA statement speaks to the formidable statistical and methodological issues involved in isolating either the effects of educators or teacher preparation programs from a complex set of factors that shape student performance.

 

“This statement draws on the leading testing, statistical, and methodological expertise in the field of education research and related sciences, and on the highest standards that guide education research and its applications in policy and practice,” said AERA Executive Director Felice J. Levine.

 

The statement addresses the challenges facing the validity of inferences from VAM, as well as specifies eight technical requirements that must be met for the use of VAM to be accurate, reliable, and valid. It cautions that these requirements cannot be met in most evaluative contexts.

 

The statement notes that, while VAM may be superior to some other models of measuring teacher impacts on student learning outcomes, “it does not mean that they are ready for use in educator or program evaluation. There are potentially serious negative consequences in the context of evaluation that can result from the use of VAM based on incomplete or flawed data, as well as from the misinterpretation or misuse of the VAM results.”

 

The statement also notes that there are promising alternatives to VAM currently in use in the United States that merit attention, including the use of teacher observation data and peer assistance and review models that provide formative and summative assessments of teaching and honor teachers’ due process rights.

 

The statement concludes: “The value of high-quality, research-based evidence cannot be over-emphasized. Ultimately, only rigorously supported inferences about the quality and effectiveness of teachers, educational leaders, and preparation programs can contribute to improved student learning.” Thus, the statement also calls for substantial investment in research on VAM and on alternative methods and models of educator and educator preparation program evaluation.

 

Related AERA Resource:
Special Issue of Educational Researcher (March 2015)—
Value Added Meets the Schools: The Effects of Using Test-Based Teacher Evaluation on the Work of Teachers and Leaders

Kenneth Zeichner and Hilary G. Conklin complain that vendors of alternative pathways into teaching have been misusing research to slam university-based teacher education. In an excerpt from a longer study, they document how organizations like Teach for America, the National Council on Teacher Quality, and the Relay “Graduate School of Education” have selectively quoted research to support their own self-interest. They seek not to improve university-based teacher education, but to replace it with entrepreneurial programs.

Zeichner is a professor of teacher education at the University of Washington, Seattle, and professor emeritus in the School of Education at the University of Wisconsin-Madison. A member of the National Academy of Education, he has done extensive research and teaching and teacher education. Conklin is a program leader and associate professor of secondary social studies at DePaul University whose research interests include teacher learning and the pedagogy of teacher education.

They write:

Critics of college and university-based teacher preparation have made many damaging claims about the programs that prepare most U.S. teachers–branding these programs as an “industry of mediocrity”–while touting the new privately-financed and- run entrepreneurial programs that are designed to replace them. These critics have constructed a narrative of failure about college and university Ed schools and a narrative of success about the entrepreneurial programs, in many cases using research evidence to support their claims.

Yet in a recent independently peer-reviewed study that will be published in Teachers College Record, we show how research has been misused in debates about the future of teacher education in the United States. Critics have labeled university teacher education programs failures and decreed their replacements successes by selectively citing research to support a particular point of view (knowledge ventriloquism), and by repeating claims based on non-existent or unvetted research, or repeatedly citing a small or unrepresentative sample of research (echo chambers).

After citing specific examples of the misuse of research, they make the following recommendations:

In order to hold all programs — public and private — to common standards of quality and evidence, we believe that several things need to be done to minimize the misuse of educational research.

First, all researchers who conduct studies that purport to offer information on the efficacy of different program models, and those who produce syntheses of studies done by others, should reveal their sources of funding, their direct and indirect links to the programs, and they should subject their work to independent and blind peer review.

Second, given that much academic research on education is inaccessible to policymakers, practitioners, and the general public, researchers should take more responsibility for communicating their findings in clear ways to various stakeholders.

Third, the media should cover claims about issues in teacher education in proportion to the strength of the evidence that stands behind them and whether or not they are supported by research that has been independently vetted.

Fourth, we should assess the quality of programs based on an analysis of a variety of costs and benefits associated with particular programs, and not just look at whose graduates can raise test scores the most. Research suggests that an emphasis only on raising test scores deepens educational inequities and continues to create a second-class system of schooling for students living in poverty.

Terry Marselle has written a research-based review of the Common Core standards. The title is: “Why the Common Core is Psychologically and Cognitively Unsound.” His book is available on amazon for FREE on Kindle from June 1-5.

Linda McNeil of Rice University has started a new blog that you should read regularly. Linda is a respected scholar of high-stakes testing and its inequitable effects. Read this paper, for example, “Avoidable Losses,” which she co-authored.

She writes:

“I want to personally invite you to visit my new blog, Educating All Our Children. This project is my response to the need I perceive for a place to bring people together around the issues of public schooling, equity and high-stakes testing. I am reaching out to you, a group that feels as passionate about equality in our schools as I do. Many of you have already shown me much support in conceptualizing this blog, for which I am very grateful.

“I have a tag line for my blog: Bringing together research, analysis, advocacy and community on behalf of the public’s schools. I’m writing in part to ask you to help me with this goal by sending me important links, leads, new stories and information that I can use to advance the movement. I would also ask that you forward, share and circulate my posts with people who share our interests. I’m not on Facebook (yet), but you can add me on Blogger and Google+ and subscribe by email to my blog.

“Thanks again for all your support over the years, and I look forward to your participation in my new venture. I hope you will visit regularly and comment often.”

Here is a sample of one of her recent posts.

Grover “Russ” Whitehurst was ousted from his job as head of the Brown Center on Education Policy at the Brookings Institution. Whitehurst had previously been research director of the Institute of Education Sciences in the George W. Bush administration. He was, of course, a big supporter of testing and choice. He even created an annual ranking of the districts with the most school choice. He gave Brookings, once known as a liberal think tank, a rightwing gloss.

In 2012, Whitehurst fired me as a Senior Fellow at Brookings, an unpaid position. I had been at Brookings since 1993. From 1993-95, I was in residence and wrote a book there on national standards. I had originally been offered the Brown Chair at Brookings but turned it down because I wanted to return to Néw York City.

Whitehurst said he was removing me from my unpaid position because I was “inactive.” At the time, my book “Death and Life of the Great American School System” was ranked as the #1 social policy book on amazon. Inactive? Hardly.

The day I was terminated, I posted on the Néw York Review of Books blog a piece that was highly critical of Mitt Romney. Whitehurst was a Romney adviser. A few hours later, I received an email from Whitehurst saying I was no longer a Senior Fellow due to my inactivity. He later said there was no connection between my anti-Romney post and his decision to drop me from a post I had held for many years, without any notice or conversation.

It will be interesting to see who replaces Whitehurst, whether liberal, conservative, neoliberal, neoconservative.

Matthew Di Carlo of the Albert Shanker Institute of the American Federation of Teachers is neither pro-TFA nor anti-TFA.

 

Here he reviews the latest study of TFA by Mathematica.

 

It has been widely reported that the study found little or no difference between the test scores of the students taught by TFA and by regular teachers. TFA saw that as a victory, since it presumably showed that no training or experience was needed to achieve the same results. Others saw it as a repudiation of TFA’s oft-repeated claims that their recruits were superior to career teachers.

 

Di Carlo parses the results and reaches this conclusion:

 

Now, on the one hand, it’s absolutely fair to use the results of this and previous TFA evaluations to suggest that we may have something to learn from TFA training and recruitment (e.g., Dobbie 2011). Like all new teachers, TFA recruits struggle at first, but they do seem to perform as well as or better than other teachers, many of whom have had considerably more experience and formal training.

 

On the other hand, as I’ve discussed before, there is also, perhaps, an implication here regarding the “type” of person we are trying to recruit into teaching. Consider that TFA recruits are the very epitome of the hard-charging, high-achieving young folks that many advocates are desperate to attract to the profession. To be clear, it is a great thing any time talented, ambitous, service-oriented young people choose teaching, and I personally think TFA deserves credit for bringing them in. Yet, no matter how you cut it, they are, at best, only modestly more effective (in raising math and reading test scores) than non-TFA teachers.

 

This reflects the fact that identifying good teachers based on pre-service characteristics is extraordinarily difficult, and the best teachers are very often not those who attended the most selective colleges or scored highly on their SATs. And yet so much of our education reform debate is about overhauling long-standing human resource policies largely to attract these high-flying young people. It follows, then, that perhaps we should be very careful not to fixate too much on an unsupported idea of the “type” of person we want to attract and what they are looking for, and instead pay a little more attention to investigating alternative observable characteristics that may prove more useful, and identifying employment conditions and work environments that maximize retention of effective teachers who are already in the classroom.

 

For me, the problem with all such studies is the assumption that the best (perhaps the only) way to identify the best teachers is by comparing changes in test scores. Great teachers supposedly get higher scores than mediocre teachers. I think that places far too much faith in standardized testing and in the assumption that education is solely measured by those tests. It makes the tests the arbiters of all things, even though most teachers do not teach tested subjects. Test-based findings are even more suspect when the children are very young.

Julian Vasquez Heilig, who recently moved from the University of Texas to California State University at Sacramento, is one of the nation’s leading authorities on Teach for America. He has studied their performance over time (see here and here), and he is not a fan. When Mathematica released its latest study of TFA, Heilig read it closely and analyzed the findings. TFA boasted that the study showed that its teachers were just as good as those who had studied education and intended to be career teachers. Some readers gleaned from this finding that “anyone can teach, no professional preparation needed,” that is, if they graduate from a highly selective college and are admitted to TFA.

 

Heilig digs deeper and has a different take on the study. The main finding, he says, is that Mathematica found no statistically significant differences in the groups of teachers they studied. However, he points out, the TFA teachers were overwhelmingly white, and few had any intention of staying in teaching as a career.

 

He notes that the test of “effectiveness” in pre-K-grade 2 is a five minute test:

 

Equally effective at what?…Mathematica utilized performance on the Woodcock Johnson III for the Pre-K-2 results— which takes 5 minutes to administer. Thus, the effectiveness of TFA teachers compared to Pre-K – 2nd grade teachers is based on a five minute administration to capture letter-word identification (Pre-K – 2) and applied problems for mathematics Pre-K – 2). Furthermore, one of the more egregious issues in the study is the aggregation of grades is that of the states that have Pre-K programs, more than half of states do not even require Pre-K teachers to have a bachelor’s degree. The report does not state that lack of a degree was an exclusion criteria and it is explicit that community preschools were included, so it appears than an aggregate that includes not only alternatively certified but also non-degreed teachers worked to TFA’s advantage. Should we really be impressed that TFA teachers outperformed a group that could have included non-degreed teachers? And they do it twice: with kindergarten and with grades K, 1, and 2.

 

What are the lessons of the study? Heilig writes:

 

So the [TFA] teachers were— on average— young, White, and from selective colleges. They had not studied early childhood in college and had very little teaching experience. They reported a similar amount of “pedagogy” (primarily the 60 hours from the five week Summer Institute), and more professional development (as we discussed above, they viewed it not very valuable). TFA teachers also reported less student teaching experience before they entered the classroom. They also were more likely to be working with a formal mentor (I mentioned David Greene’s point about the drain on mentors due to the constant carousel of Teach For America teachers in and out of schools here). As new teachers, they spent more time planning their own lessons, but were less likely to to help other teachers. Finally, TFA teachers were less satisfied “with many aspects of teaching” and less likely to “plan to spend the rest of the career as a classroom teacher….”

 

In conclusion, read at face value, here is the message Mathematica appears to promulgate with the report:

 

We do not need experienced (read: more expensive) teachers when non-experienced, less expensive teachers get the “same” —though not statistically significant— outcomes.
We do not need a more diverse workforce of teachers, again, because TFA teachers, who are overwhelmingly white, get the same outcomes.
Is TFA really in alignment with a vision for providing every student a high quality teacher? Or do they, Mathematica et al. just keep telling us that they are?

 

For myself, I have read many times that Teach for America invites young people to “make history” by serving for two years. And Wendy Kopp has frequently said that “One day,” all children in America will have an excellent teacher. I have a hard time understanding the logic of these claims. If the TFA teachers get the same results as current teachers, how is that “making history”? If most TFA recruits leave after two years, how does that lead to the conclusion that one day all children will have an excellent teacher? If TFA persuades policymakers that teachers can do a good enough job with no professional preparation, doesn’t that decimate the idea of teaching as a profession? If anyone can teach so long as they went to a selective college, how does that raise the standard for teachers? If our policymakers prefer churn, with teachers leaving every two or three years to find their real career, how is that good for students? How does TFA improve the profession? It doesn’t. It eliminates it.

 

For his fearlessness, for his willingness to stand up to those with money and power, for his willingness to present the evidence as he finds it without fear or favor, I place Julian Vasquez Heilig on the honor roll of this blog. He is an example to all researchers of the ethics of his profession. To be an outstanding researcher requires years of study, scholarship, discipline, dedication, and experience. Sort of like being a great teacher.