Archives for category: Research
« Older Entries
Newer Entries »

Betts and Tang Respond to Lopez on Charter Study

By dianeravitch
November 11, 2014 //
0

On September 30, Francesca Lopez of the University of Arizona reviewed a study of charter schools by Julian R. Betts and Y. Emily Tang of the University of California at San San Diego

Betts and Tang here respond to Lopez’ critique of their study of charter school achievement..

The critical study by Lopez was published by the National Education Policy Center and posted on this blog.

Betts and Tang say that Lopez misrepresented their study. They write:

“First, what did we find in our meta-analysis of the charter school effectiveness literature? On average, charter school studies are revealing a positive and statistically significant difference between math achievement at charter schools and traditional public schools. We also find a positive difference for reading achievement, but this difference is not statistically significant. Second, we devote much of our paper to studying not the mean effect, but the variation across studies in the effect of attending a charter school. We find that charter schools’ effectiveness compared to nearby traditional public schools varies substantially across locations.

“What is the central claim of Lopez? She writes: “The report does a solid job describing the methodological limitations of the studies reviewed, then seemingly forgets those limits in the analysis” (p. 1). She uses words like “exaggeration” and “overstated” (p. 8) to characterize our analysis of the literature, and implies that our conclusions are not “reserved,” “responsible,” (p. 7) or “honest” (p. 7 and p. 8).

“Throughout her essay, Lopez falsely projects intentions in our words that simply are not there. We encourage interested readers to review the words that we actually wrote, in their full context, in our abstract, main paper, and our conclusion. We are confident that readers will confirm for themselves that any “overstated” conclusions of which Lopez accuses us are imagined.”

“There are serious problems with Lopez’s arguments. First, she habitually quotes our work in a selective and misleading way. Such rhetorical slights, in which she quotes one of our sentences while ignoring the highly relevant adjacent sentences, or even cutting important words out of our sentences, overlook important parts of our analysis and result in a highly inaccurate presentation of our work. Second, her analysis contains six technical errors. These technical mistakes, some quite serious, invalidate many of Professor Lopez’s claims. An appendix to this essay exposes more than two dozen misleading or outright incorrect statements that Lopez makes in a mere 9-page essay. To give readers a sense of the scope and severity of these problems, consider the following examples:

“Example 1: A Partial and Misleading Quotation

“Lopez insinuates that we exaggerate the positive overall math effect while downplaying the percentage of studies that show negative results. She writes:
“The authors conclude that ‘charter schools appear to be serving students well, and better in math than in reading’ (p. 47) even though the report finds ‘…that a substantial portion of studies that combine elementary and middle school students do find significantly negative results in both reading and math—35 percent of reading estimates are significantly negative, and 40 percent of math estimates are significantly negative (p. 47)’”

“Here is what we actually wrote on page 47: “Examining all of these results as separate parts of a whole, we conclude that, overall, charter schools appear to be serving students well, and better in math than in reading. The caveat here is that a substantial portion of studies that combine elementary and middle school students do find significantly negative results in both reading and math—35 percent of reading estimates are significantly negative, and 40 percent of math estimates are significantly negative.”

“Lopez uses two rhetorical devices to lead readers to the perception that we overstated findings. First, she separates the two quotations, implying that we are somehow hiding the second result, when in fact we intentionally mention the positive overall mean math effect and the variation in the results across studies side by side. Second, she further misleads the reader by again cutting out part of our sentence. Instead of stating that we have a “caveat” to the positive mean math effect she removes that entire clause.

“What makes the approach of Lopez even more misleading is that in the paragraph above, we were bending over backwards to be fair. We cite only one type of study in that quotation: those that combine elementary and middle schools. (These account for about 1/7th of all the studies.) Why did we focus only on those studies in the above quotation? Because these studies were the exception to our conclusion—the ones that produced the highest percentage of studies with negative and significant estimates. Wouldn’t one think that if our goal had been to overstate the positive effects of charter schools we would never have chosen to list the result that is the least favorable to charter schools in the text above? For example, we could have stated that for elementary school studies, only 12% showed negative and significant reading results, compared to 71% showing positive and significant results. Or we could have stated that only 11% of elementary school studies showed negative and significant math results, while 61% showed positive and significant results in math.

“Lopez fails to list any of the more positive results from the other grade span categories studied that led us to our overall conclusion. We noted the exception above precisely because it was an exception. While it is worth noting, it does not refute the other evidence. By citing an exception as a reason to dismiss all of the other results, Lopez misses the main point of a statistical meta-analysis. This is a consistent pattern throughout her essay.”

Betts and Tang make 26 points about the flaws of Lopez’s analysis.

Categories Charter Schools, Research

Néw York Regents: Test Scores Should Be 40% of Teacher Evaluation

By dianeravitch
August 26, 2014 //
48

Remember that Arne Duncan said that there was too much testing, that testing was sucking the oxygen and joy out of classrooms? New York didn’t get the message. In that state, state tests count for 20% of educator evaluations, and local assessments count for another 20%. That is the agreement negotiated with the unions when the state won Race to the Top funding.

That was then, this is now.

The Néw York Board of Regents want state test scores to count for 40% of the evaluations of teachers and principals. This report was was confirmed to me by someone in Albany.

It matters not to the Regents that test-based evaluation is not working anywhere else. It matters not that the AERA and the National Academy of Education warned against it, warned that it would incentivize teachers to avoid high-needs students. It matters not that the American Statistical Association warned against using test scores to rate individual teachers since they affect only 1-14% of variation in student scores.

The ASA said: “Attaching too much importance to a single item of quantitative information is counter-
productive—in fact, it can be detrimental to the goal of improving quality. In particular, making changes in response to aspects of quantitative information that are actually random variation can increase the overall variability of the system.”

Unlike the state of Vermont, which refuses to rate teachers and principals by test scores, Néw York’s Regents will plunge ahead, regardless of the damage they do to teachers, principals, students, and communities.

Categories New York, Research, Teacher Evaluations, Testing

Andy Hargreaves: What Is the Purpose of Educational Benchmarking?

By dianeravitch
August 25, 2014 //
21

Andy Hargreaves of Boston College asks an important question: What is the purpose of benchmarking? We collect data, we measure, we test, we set goals, but why? Will it improve performance if we know that someone else does it better? Do they have the same challenges, the same resources? Is there more to education than raising tests ores and do higher test scores necessarily mean better education?

Andy begins with two stories about benchmarking, one positive, one negative. One improved public health, one made it easier to conduct war.

Right now, under pressure from No Child Left Behind and Race to the Top, everything is measured. Why? To fire teachers and principals? To close schools? To hand public property to entrepreneurs? Who benefits? What do we do with the losers? Throw them away? Plenty of children were left behind, and many will not make it to “the top.”

Andy writes:

“Is the purpose of our educational benchmarking to further the public good, to raise the standards of education for all, to elevate the poorest and most disadvantaged students to the greatest heights of accomplishment? And once we have done our calculations and made our maps, what pathways will be opened up, and what people and resources will be pulled along them in this worthy quest for equity and excellence? The White House announced earlier this summer that it would address educational inequities by collecting data to help pinpoint where they existed, but there seemed to be no plan to bring up the people and resources to correct them.

“Is there a second purpose of educational benchmarking then? Is it to delineate the weak from the strong, inciting nation to compete against nation, Americans against Asians, and school against school. After we have pinpointed schools that are failing, does this just make it easier for invading opportunists to set up charter schools in their place, or to market online alternatives, tutoring services and the like?

“As in surveying, benchmarking in education should be about discovering where we stand and learning about who we are and what we do by observing those around us. It should be about improving public education, just as the sewer maps for my hometown contributed to public sanitation. Benchmarking should not be about fomenting panics about performance in relation to overseas competitors. And it should not be about dividing schools, families and communities from each other to create easy pickings for the educational market.

“Whenever we are engaged in the data-driven detail of educational benchmarking, these are the greater questions we should be asking. Of what map or whose map are we the servants?”

Categories Closing schools, Data and Data Mining, International, Research

Columbia Scholar Says Chetty, Friedman, and Rockoff Are Wrong About VAM

By dianeravitch
July 28, 2014 //
85

Moshe Adler, a professor at Columbia University, has emerged as one of the most incisive critics of the work of Raj Chetty, John Friedman, and Jonah Rockoff on Value-added measurement (VAM).

In the recent Vergara decision about tenure for teachers in California, the study by Raj Chetty and John Friedman of Harvard and Jonah Rockoff of Columbia played a prominent role. But according to the economist Moshe Adler the study is wrong and misleading. According to Adler, the authors suppressed a result that contradicts their main claim, they picked and chose which data sets to use, they used a misleading method of analysis that inflated their results and they misrepresented research that contradicts their claims as supporting them. These are just a few of the problems with the scientific integrity with the study. Adler wrote his review for the National Education Policy Center and it can found at: http://nepc.colorado.edu/newsletter/2014/06/adler-response-to-chetty)

A short time after the publication of his NEPC review, Adler received an email from Chetty that informed him that the study had been accepted for publication by the American Economic Review (AER). (Chetty also suggested that a Nobel Prize will likely follow!) Adler immediately wrote to the editors of the AER to alert them to the grave scientific problems with the study. The editor-in-charge did not evaluate Adler’s objections herself, nor did she send them to the referees to evaluate. Instead, she forwarded Adler’s letter to the authors who then replied to Adler’s NEPC review. The editor found this reply satisfactory, but as Adler explains in his response, Chetty’s et al.’s reply is without merit, and it only adds to the problems with the research. Chetty’s letter to Adler and Adler’s correspondence with the AER can be found at: http://www.columbia.edu/~ma820/Chetty%20Adler%20AER.htm

Categories Research, Teacher Evaluations, Teacher Tenure

Jack Hassard: NCTQ Ratings Are “Junk Science”

By dianeravitch
June 28, 2014 //
7

Jack Hassard, professor emeritus of science education at Georgia State University, here reviews the ratings of the National Council on Teacher Quality and declares them to be “junk science.” He looks at the Georgia institutions of teacher preparation and finds that the ratings are haphazard, spotty, and inaccurate. The he gathers some of the major critiques by others and concludes that the ratings as a whole are bogus, nothing more than propaganda to undermine teacher preparation and force it into NCTQ’s political framework. He calls the NCTQ ratings an “assault on teacher preparation.”

Professor Hassard taught science education at GSU for 33 years. He notes that 21 institutions offer 269 programs for teacher preparation in Georgia. Of those 269, the NCTQ reviewed 39, not by visiting them but by reading course catalogues and syllabi, which reveal nothing about the quality of the programs. He calls the Georgia ratings “feeble and incompetent.”

The ratings were assembled, writes Hassard, by unqualified reviewers: “We analyzed the make-up of the NCTQ people, and discovered that it represents a “stacked deck.” Only 2.5% of the participants in the review were teacher educators–active professors out there doing teacher education. The NCTQ was stacked with corporate executives, foundation executives, and employees of NCTQ. It was far from representing the field of teacher education.”

He adds: “The “methods” used include sources including: syllabi (when they can get them), textbooks, catalogs, handbooks, evaluation forms. We show that the NCTQ report on teacher preparation is junk science. The method that they employed in their study avoided data from the very sources that could help uncover the nature of teacher preparation. These sources are faculty, administrators, students, and cooperating school districts and educators. Without interviewing and observing teacher preparation programs directly, and without establishing a cooperative relationship with the these institutions, the NCTQ condemns itself to false claims, outright opinions that have little bearing on the nature of teacher preparation.”

Categories National Council on Teacher Quality (NCTQ), Research

Beardsley: Another Critic of VAM & Chetty et al

By dianeravitch
June 6, 2014 //
12

Audrey Amrein Beardsley invited an economist to review Raj Chetty & Co’s effort to take down the statement of the American Statistical Assosociation.

Chetty and friends are the leading advocates for using test scores to rank teachers and fire those whose students have the lowest scores. The ASA report was inconvenient for their thesis, as it pointed out that teachers account for a small percentage of the variance in test scores, that what is observed is likely to be correlation all, not causal, and that putting so much weight on test scores was likely to cause bad effects.

There is also the inconvenient fact that VAM is no longer a neat theory, but has moved into the realm of reality. Some districts have used it, often with unfortunate results. One would think that academics would feel some obligation to see how their theory is working in practice rather than continue to market it on grounds of its theoretical elegance.

Categories Research, Teacher Evaluations, Testing

Peter Greene: Why “Research” Doesn’t Tell Us Whether a School is Working

By dianeravitch
May 25, 2014 //
21

Peter Greene steps into a debate about whether schools are “working.”

One answer: let research tell us.

Greene disagrees.

Research, he says, depends on the questions you consider important. In the past, communities decided what they wanted their schools to do.

He writes:

“Because what works and what doesn’t work is not a matter of good research at all. Or rather, the research doesn’t matter.

“Only one thing matters– the definition of “works.”

“Does this raggedy philips head screwdriver work? That depends on whether I want to use it to unscrew screws or punch holes in a soup can. Does telling my wife she’s fat work? That depends on whether I want to make her happy or angry.

“If I get to define what “working” looks like, all the measuring, testing, researching, test tubial navalgazing introexamination that follows is secondary. Part of what gets folks’ backs up about the Reformsters is that they start with, “You do not understand how a school is supposed to work. You are doing school wrong.”

Well, I don’t agree with Peter that research doesn’t matter. I think that the corporate reformers choose whatever research fits their policies. If there is none, then they ignore research. They know what they want to do, and no research will change their minds. They decide that “no child should be left behind” and ignore research showing that such a utopia never happened without kicking kids out or otherwise gaming the system. They say that teachers should be rated by the scores of their students but when research says that’s a truly stupid idea, they ignore research. When they boast about the glories of vouchers, charters, and competition, they conveniently ignore the Petri dish of Milwaukee, where this combination has been a disaster.

Categories Closing schools, Research

Does Data Matter to the Status Quo Defenders?

By dianeravitch
April 24, 2014 //
83

Reader Chiara Duggan says that study after study shows that charters and vouchers demonstrate that data don’t change their minds. She is right. The charters that get high test scores systematically exclude the most challenging students. Some public schools get higher test scores because they serve affluent districts. The differences between charters, vouchers, and public schools tend to be small if they enroll the same students. But the Status a quo pays large numbers of people to argue that the Status Quo–the destruction of an essential institution of a democratic society–is “working” and has positive effects. When the test scores don’t support their argument, they shift the goal post and claim that the private schools–the charters and vouchers–have higher graduation rates. They take care not to mention attrition rates, which are very high. In the case of Milwaukee, the “independent” evaluators from the Walton-funded University of Arkansas quiet.y acknowledged that 56% of those who started in voucher schools left before graduation.

Chiara writes:

Oh, data doesn’t matter to ed reformers. It’s a belief system. Private is better than public. You can’t move someone off a belief with numbers.

How many times have you see a voucher study like this over the years? Once a year for two decades? Yet Democrats and Republicans and paid lobbyists and pundits still promote publicly-funded private schools over public schools. Vouchers have expanded every single year in this country under ed reformers. There isn’t a scintilla of evidence that they’re any better than the public schools they undermine and then replace, but it simply doesn’t matter.

“Students attending private schools receiving taxpayer-funded vouchers in a new statewide program did not score as high overall as public school students on state tests in reading and math, according to data released Tuesday by the Department of Public Instruction.”

It doesn’t matter what public schools do; improve, don’t improve, whatever. They are the designated punching bags for the punditry set. It’s knee-jerk at this point. Heck, a lot of people are PAID to bash them. It’s a smart career move.

I think this may inadvertently benefit public school students. As it becomes more and more clear that privately-run schools don’t outscore public schools in any meaningful way, the goalposts will move, and standardized test scores will no longer be the measure. I think it’s already happening. Ed reformers may actually do something that benefits public schools, and deemphasize the lunatic, obsessive fealty to test scores. They’ll do it it only to defend their own schools, but public schools may benefit collaterally.

Read more: http://host.madison.com/news/local/education/blog/dpi-wisconsin-voucher-schools-show-lower-test-scores-compared-to/article_df494180-cd29-538a-80be-a923cded39aa.html#ixzz2yNzhk7yP

–

Categories Accountability, Charter Schools, Corporate Reformers, Data and Data Mining, Research, Vouchers

Carol Burris: Please DON’T Stay the Course

By dianeravitch
April 19, 2014 //
17

Award-winning high school principal Carol Burris reports here on Arne Duncan’s latest foray into New York, where he highly praised the state’s controversial Commissioner of Education John King, disparaged disgruntled educators and parents as a mere distraction, and urged the state to “stay the course.”

Burris, a leader in the effort to expose and reverse some of the worst aspects of Race to the Top, explains why it is important not to stay the course, when the course is leading in the wrong direction.

She writes:

” There is no empirical evidence that rigorous state or national standards will result in higher student achievement or greater college readiness.

“Those who created the Common Core assumed that if we established rigorous standards, student achievement and economic competitiveness would increase. Duncan said, in his remarks at New York University, that it is common sense. Prior to the 15th century, common sense said the world was flat, but that did not make it true.”

She cites research to demonstrate that rigorous standards and high-stakes tests o not produce better education:

“This is not an argument for low standards or no standards—it is an argument that standards reform is not an effective driver of school improvement. Keep in mind that all state standards had high-stakes state tests associated with them. The more rigorous the standards, the more difficult the tests are. As high-stakes tests become more difficult, the curriculum becomes narrower and narrower. The tests soon drive teaching and learning.

“When I hear “I am for the Common Core standards, I am just not for the tests”, I cringe. While thoughtful educators look at the standards through their prism of good practice, test makers look at the standards as the basis for creating “items” that discriminate the learning of one child from another. In the end, the test maker calls the shots. It is no coincidence that the Common Core Standards, PARCC and Smarter Balanced were all born at the same time. In his remarks, Duncan referred to PARCC and Smarter Balanced as the “national tests.”

“The destination of school reform—ensuring that all students have the skills, content and habits needed for college and career success—is the right destination. The challenge is choosing the pathway that gets us there. Good intentions are not enough. If we continue to put our tax dollars and our efforts into “standards reform” because Mr. Duncan and his followers believe it is common sense, we will waste time and treasure.”

Bottom line: Race to the Top is no better than No Child Left Behind. It has no research to support its premises and will come to an ignominious end like its predecessor. Burris hopes that Duncan will change course but his bad ideas seem impervious to evidence.

Categories Duncan, Arne, Race to the Top, Research, Testing, U.S. Department of Education

Laura H. Chapman: The Desperate Search for a Science of Teacher Evaluation

By dianeravitch
April 18, 2014 //
14

Laura H. Chapman left the following comment. The word “desperate” to describe this quest for a scientific, data-based means of judging teachers is mine. Something about it smacks of anti-intellectualism, the kind of busywork exercise that an engineer would design, especially if he had never taught K-12. This is the sort of made-up activity that steals time from teaching and ultimately consumes a lot of time with minimal rewards.

Chapman writes:

Please give at least equal attention to the 70% of teachers who have job assignments without VAMs (no state-wide tests). For this majority, USDE promotes Student Learning Objectives (SLOs) or Student Growth Objectives (SGOs), a version of 1950s management-by-objectives on steroids.

Teachers who have job-alike assignments fill in a template to describe an extended unit or course they will teach. A trained evaluator rates the SLO/SGO (e.g. “high quality” to “unacceptable” or “incomplete”).

The template requires the teacher to meet about 25 criteria, including a prediction of the pre-test to post-test gains in test scores of their students on an approved district-wide test. Districts may specify a minimum threshold for these gains.

Teachers use the same template to enter the pre-and post-test scores. An algorithm determines if the gain meets the district threshold for expectations, then stack ranks teachers as average, above or below average, or exceeding expectations.

1. The Denver SLO/SGO template is used in many states. This example is for art teachers—-Denver Public Schools. (2013). Welcome to student growth objectives: New rubrics with ratings. http://sgoinfo.dpsk12.org/
2. One of the first attempts to justify the use of SLOs/SGOs for RttT—-Southwest Comprehensive Center at WestEd (n.d.). Measuring student growth in non-tested grades and subjects: A primer. Phoenix, AZ: Author. http://nassauboces.org/cms/lib5/NY18000988/Centricity/Domain/156/NTS__PRIMER_FINAL.pdf

3. This USDE review shows that SLOs/SGOs have no solid research to support their use—-Gill, B., Bruch, J., & Booker, K. (2013). Using alternative student growth measures for evaluating teacher performance: What the literature says. (REL 2013–002). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic. http://ies.ed.gov/ncee/edlabs.

4. The USDE marketing program on behalf of SLOs/SGOs—-Reform Support Network. (2012, December). A quality control toolkit for student learning objectives. http://www2.ed.gov/about/inits/ed/implementation-support-unit/tech-assist/slo-toolkit.pd

5. The USDE marketing campaign for RttT teacher evaluation and need for district “communication SWAT teams” (p. 9) —- Reform Support Network. (2012, December). Engaging educators, Toward a New grammar and framework for educator engagement. Author. http://www2.ed.gov/about/inits/ed/implementation-support-unit/tech-assist/engaging-educators.pdf

6. Current uses of SLOs/SGOs by state—-Lacireno-Paquet, N., Morgan, C., & Mello, D. (2014). How states use student learning objectives in teacher evaluation systems: a review of state websites. Washington, DC: US Department of Education, Institute of Education Sciences. http://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/REL_2014013.pdf

7. Flaws in the concepts of “grade-level expectation” and “a year’s worth of growth” —-Ligon, G. D. (2009). The optimal reference guide: Performing on grade level and making a year’s growth: Muddled definitions and expectations, growth model series, Part III. Austin, TX: ESP Solutions http://www.espsolutionsgroup.com/espweb/assets/files/ESP_Performing_on_Grade_Level_ORG.pdf

Categories Research, Teacher Evaluations, U.S. Department of Education
« Older Entries
Newer Entries »