How many times have you read in a report or in the newspaper that X method or Y school was able to produce an extra 40 days or extra weeks of learning in reading or math?

 

How do gains in test scores get converted into days or weeks or months?

 

The answer, according to Gary Rubinstein, is that they don’t. Or they shouldn’t. It is nonsense.

 

I recently read a Mathematica Policy Research report on the Teacher Incentive Fund (merit pay), which claimed that a 1% increase in test scores was equivalent to an additional three weeks of learning. See here (study snapshot) and here (executive summary) and here (full report).

 
Performance Bonuses for Educators Led to Small Improvements in
Student Achievement

 

Educators’ understanding of bonus program improved, but challenges remain

 

New findings from Mathematica Policy Research show that a federal program providing bonuses to educators based on their performance had a small, positive impact on student achievement. In the first report to describe the effects of pay-for-performance bonuses within the Teacher Incentive Fund (TIF) program on student achievement, researchers found that student scores on standardized reading tests rose by 1 percentile point—the equivalent of about three weeks of additional learning. The study also showed similarly positive, but statistically insignificant, improvements in math.

 

I asked Gary if it made sense to translate a one-point gain into three weeks of learning, and he replied:

 

Mathematica should stop using that ‘weeks of learning’ metric. They use a calculation that says that average teachers don’t teach very much so that they maybe get the kids to increase their scores from 24 percent passing (if they did not teach anything) to 34 percent in the entire year. So each ‘point’, by that logic, amounts to about a sixth of the year. A teacher with merit pay, then who gets that extra ‘point’ would be teaching 10% more in that year which is an extra three weeks. I wish they would just give the raw score which people could relate to, like there were 50 questions on the test and students of people without merit pay got 25 correct and students of people with merit pay got 26 correct. Then people would be able to put these numbers into perspective and realize that they are not a big deal.