Archives for the year of: 2014

Audrey Amrein-Beardsley has updated her reading lists on value-added assessment. Most of the studies cited show that it is inaccurate, unstable, and unreliable. The error rate is high. Students are not randomly assigned to teachers. Ratings fluctuate from year-to-year. About 70% of teachers do not teach tested courses. Perhaps that is why other nations do not judge teachers by the rise or fall of the test scores of their students. Unfortunately in this country, at this time, we have a cult worship of standardized testing, which is used to evaluate students, teachers, principals, and schools. People’s lives hang on the right answer. In a just world this practice would be recognized for what it is: Junk science.

Here are her top 15 studies. Open the link to find the top 25. Open the link to find links for all these readings. With Beardsley’s help, you too can be an expert.

American Statistical Association (2014). ASA statement on using value-added models for educational assessment. Alexandria, VA.

Amrein-Beardsley, A. (2008). Methodological concerns about the Education Value-Added Assessment System (EVAAS). Educational Researcher, 37(2), 65-75. doi: 10.3102/0013189X08316420.

Amrein-Beardsley, A., & Collins, C. (2012). The SAS Education Value-Added Assessment System (SAS® EVAAS®) in the Houston Independent School District (HISD): Intended and unintended consequences. Education Policy Analysis Archives, 20(12), 1-36.

Baker, E. L., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn, R. L., Ravitch, D., Rothstein, R., Shavelson, R. J., & Shepard, L. A. (2010). Problems with the use of student test scores to evaluate teachers. Washington, D.C.: Economic Policy Institute.

Baker, B. D., Oluwole, J. O., & Green, P. C. (2013). The legal consequences of mandating high stakes decisions based on low quality information: Teacher evaluation in the Race-to-the-Top era. Education Policy Analysis Archives, 21(5), 1-71.

Darling-Hammond, L., Amrein-Beardsley, A., Haertel, E., & Rothstein, J. (2012). Evaluating teacher evaluation. Phi Delta Kappan, 93(6), 8-15.

Fryer, R. G. (2013). Teacher incentives and student achievement: Evidence from New York City Public Schools. Journal of Labor Economics, 31(2), 373-407.

Haertel, E. H. (2013). Reliability and validity of inferences about teachers based on student test scores. Princeton, NJ: Education Testing Service.

Hill, H. C., Kapitula, L., & Umland, K. (2011). A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal, 48(3), 794-831. doi:10.3102/0002831210387916

Jackson, C. K. (2012). Teacher quality at the high-school level: The importance of accounting for tracks. Cambridge, MA: The National Bureau of Economic Research.

Newton, X., Darling-Hammond, L., Haertel, E., & Thomas, E. (2010). Value-added modeling of teacher effectiveness: An exploration of stability across models and contexts. Educational Policy Analysis Archives, 18(23), 1-27.

Papay, J. P. (2010). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48(1), 163-193. doi:10.3102/0002831210362589

Paufler, N. A. & Amrein-Beardsley, A. (2014). The random assignment of students into elementary classrooms: Implications for value-added analyses and interpretations. American Educational Research Journal, 51(2), 328-362. doi: 10.3102/0002831213508299

Rothstein, J. (2009). Student sorting and bias in value-added estimation: Selection on observables and unobservables. Education Finance and Policy, 4(4), 537-571. doi: http://dx.doi.org/10.1162/edfp.2009.4.4.537

Schochet, P. Z. & Chiang, H. S. (2010). Error rates in measuring teacher and school performance based on student test score gains. Washington DC: U.S. Department of Education.

A study commissioned by school leaders in New York’s Lower Hudson Valley reviewed the state’s teacher evaluation system and concluded that it was irreparably flawed.

“The study, released Friday, found that the state formula for calculating evaluations forces school districts to inflate classroom-observation ratings so teachers do not get poor overall scores.

“If districts were to give more accurate grades to teachers after classroom visits, the study found, many teachers would “unjustly” receive overall ratings of “developing” or “ineffective.” Such districts would “end up looking like they have an underperforming workforce,” the report said.

“This is not something that can be fixed; the state Education Department needs to start over,” said Louis Wool, Harrison schools superintendent, who was president of the Lower Hudson Council of School Superintendents when the group commissioned the study last year.

“The study reviewed 2012-13 evaluation results for 1,400 teachers in 32 districts in Westchester, Rockland, Putnam and Dutchess counties. The superintendents group provided the data to Education Analytics, a non-profit organization in Madison, Wisconsin, which did the study.

“Researchers credited New York state with improving its methods of measuring teacher effectiveness. In fact, the report called New York “a pioneer” in developing a modern evaluation system. But researchers said there are few examples nationally of effective implementation and that strong use of data may not necessarily translate into good policy.”

The state apparently wants a system that gives many teachers low scores so they can be fired; schools and districts want to retain their decision-making power over which teachers should be kept or terminated. The state is trying to take that authority away from schools and districts by creating a mechanical formula. The formula doesn’t work, and no such formula works anywhere in the country. The biggest problem in teaching today is recruiting, supporting, and retaining good teachers, not finding and firing bad ones. Any administrator worth her salt knows how to do the firing part.

The state should not start over. The state should get out of the way.

Peter Greene has a very engaging post about the insanity of Marshall Tuck’s run for State Superintendent of California.

 

Greene can’t believe that Tuck believes what he is saying and promising. If he delivers, he will destroy public education in California, get rid of experienced teachers, somehow find inexperienced replacements for them, and then what? Then Californians will know that the whole reformster agenda is a fraud. Maybe, just maybe, Greene thinks, it would be a good thing to have this expose happen.

 

What qualifies Tuck to run the state education department? Well, he was an investment banker. The rich and powerful like him. He has friends in Hollywood. He thinks no teacher should have tenure. He failed as leader of Green Dot. He failed running the mayor’s takeover schools. That means he is an expert on reform.

 

Greene writes:

 

Tuck is popular with the Let’s Kick Teachers’ Asses crowd, which is why this election matters. Current Superintendent Tom Torlakson pissed off a lot of powerful people by deciding to challenge the Vergara ruling, and if elected Tuck will put an end to that toot suite.

 

I confess to being a little fascinated by the Tuck candidacy, because what is the end game here? I mean, unless he’s an idiot, he has to know that the same smoke and mirrors that create the illusion of success for charter schools cannot be scaled to the state level, and his bold claims that he can raise California’s educational standings will fail hugely. “Throw out difficult students who make school look less successful” only works if there are other schools to send them to. Maybe he has figured out how to scale charter success with, say, a plan to push all low-performing California students into Nevada. But I’m doubtful. He has to know that he cannot deliver any of the results he is promising.

 

So if he’s not an idiot, what’s the plan here? Just get in there and strip as much money as possible out of the system and walk away? Destroy the teaching profession and public education and just hope nobody notices or cares? The usual reformster profile is to find yourself a job where you aren’t accountable to much of anybody and where the reporting of results is entirely under your control. But Tuck wants to be responsible to the state voters for an entire state system whose results will be pretty hard to hide.

 

If this guy is elected, shame on the voters of California. Their children will get what they don’t deserve.

 

 

This curious story caught my eye as I was reading the business section of the New York Times yesterday. A man named Jesse Busk is suing Amazon because, after his 12-hour shift at its warehouse in Las Vegas, he was required to wait in line for a security screening along with other workers to see if they had stolen any goods. Busk maintained that he should be paid for the additional 25 minutes because it was a required part of his workday. His employer and other employers obviously were opposed to paying him for the extra time.

Mr. Busk has since been laid off. He said it was unfair not to be paid for the extra time after such a long day.

Two things got my attention in this story. First, I thought this country had long ago accepted that a man or woman should. Not work more than eight hours a day, forty hours a week. I was wrong. As I continued reading about the rationales for and against Mr. Busk’s contention, I discovered that the Obama administration was supporting employers, not the worker. Naturally, I found myself wondering what would Harry Truman or FDR or JFK have done. Which side would they be on.

Arne Duncan issued waivers to 43 states to allow them to avoid the sanctions of the No Child Left Behind Law, passed in 2001, signed into law in January 2002. NCLB is an utter disaster, recognized as such by everyone except the people who had a direct hand in writing it. It requires that 100% of all children in grades 3-8 must be “proficient” on state tests of reading and mathematics or the school will face dire consequences.

 

In no nation in the world are 100% of all children proficient in reading and math. Congress’s mandate was a cruel joke on the nation’s public schools.

 

In order to get Duncan’s waiver, states had to agree to Duncan’s terms. One of them was that the state had to create a teacher evaluation system based on test scores. Washington State initially agreed, but as the research accumulated showing that this strategy was not working anywhere, the legislature refused to pass such a system.

 

Duncan revoked the waiver he had in his lordly manner extended. Now almost every school in the state is a failing school and must spent at least 20% of their federal funding on private tutoring or allow students to transfer to “non-failing” schools, if they can find one.

 

This article by Motoko Rich in the New York Times shows the ugly consequences of Duncan’s policies have been on the public schools of Washington State. Schools that have shown dramatic improvement in recent years are now declared failures. Duncan says the state must suffer the consequences of its failure to follow his orders.

 

This man is not fit to be Secretary of Education. He is a promoter of privatization and high-stakes testing. His period in office has been marked by massive demoralization of teachers and educational stagnation (his own term). From his actions, it appears that he doesn’t care for public education and hopes it will be replaced by privately managed charters and vouchers. His action in this case has caused harm to the students and teachers of Washington State. The headline of the article says he put schools “in a bind.” It would be more accurate to say that Duncan has rained chaos on the schools and children of Washington State. The sooner he is out of office, the sooner we can turn to realistic ways of helping children and schools.

By a vote of 3-2, the school board of Jefferson County, Colorado, passed its controversial proposal to adopt an American history curriculum that removes references to dissent and social disorder and anything else that diminishes a sense of patriotism. This idea was cooked up by a radical rightwing majority that took control of the board at the last election.

The meeting was noisy and fractious. Students turned out in large numbers to oppose the sanitized curriculum, and by their actions, showed that dissent is alive and well.

Luckily, there is a website devoted to watching the JeffCo school board. The Jeffcoschoolboardwatch says the word of the day now is: Recall!

The students have gained national and international attention. The school board majority and its allies say they are “pawns” of the teachers’ union. Fox News called them “punks.”

Peter Dreier, a professor at Occidental College in California, proposes that the major historical associations honor these students for demanding a history curriculum that is not saddled with ideological bias. They have stood up for academic freedom.

I call them heroes. Students cannot be fired. They can stand up for their right to learn and for their teachers’ right to teach. Teachers and principals can’t do that. Student protests can awaken the public. They can alert the people of JeffCo and Colorado about radical efforts to remove controversy from the teaching of U.S. history. They can save our schools from the reactionaries who want to hand them over to Walton-funded, Broad-funded, Gates-funded, NewSchools Venture-funded profiteers. They can stop data-mining.

Their voices cannot be stilled by threats and intimidation. They have the idealism of youth and the freedom to act and speak without fear. Go, students of JeffCo!

Thanks to Pam Hricik for sending this gem of a question.

Read this and evaluate Mr. Jackson.

Ok, I’m a sucker for dog videos. Especially if they make me laugh. There is this guy named Nic, and he has a wonderful chihuahua named Pancho. There are many Nic and Pancho videos, all very short. This is the funniest.

I have a big mutt named Mitzi. She is part German Shepherd and part many other breeds. She is not trained to do anything. But she is very lovable and sweet. When people see her, they are frightened, because she is big, about 65 pounds. But she doesn’t have an aggressive bone in her body.

In this mini-essay, left as a comment, Bob Shepherd notes that Common Core testing assumes that there is only one correct answer when interpreting literature. This, he says, is a complete rejection of reader-response theory, which had been prevalent for many years. Shepherd has many years of experience writing curriculum, assessments, and textbooks.

He writes:

“Years ago, I was doing a project for one of the major textbook publishers—writing for a high-school British literature textbook. I was given an assignment to write a lesson on Robert Burns’s poem “A Red, Red Rose.” This poem begins, you may remember, with the following lines:

O my Luve’s like a red, red rose
That’s newly sprung in June;
O my Luve’s like the melodie
That’s sweetly play’d in tune.

One of the questions that I asked about the poem was, “Why does the speaker compare his beloved to a red, red rose?” And the answer I wrote for the answer key was something like, “The speaker wishes to communicate that his this person is attractive and that he loves her, and so he compares her to a red rose, which is a traditional symbol of beauty and of romantic love.” I could have elaborated: Probably through association with blood and with blushing, the color red traditionally symbolizes intense emotion, or ardor. Roses are attractive and share this property with the objects of romantic affection. For these reasons, it became conventional to speak of someone as being “a red rose” in order to communicate that a) she (or, more rarely, he) was beautiful and b) that she (or, more rarely, he) was an object of ardent emotion, and c) that that emotion was one of romantic attraction. The speaker is therefore using a conventional symbol.

I could have added that the reason why the poet chose to express this in a simile rather than in a metaphor (“O my Luve’s a red, red rose”) was probably as mundane as to fill out the meter. I could further have explained that it is the beloved not the speaker’s feeling that is compared to a rose, for later in the poem, the speaker uses the same word, Luve, in direct address:

And fare thee well, my only Luve
And fare thee well, a while!
And I will come again, my Luve,
Tho’ it were ten thousand mile.

The editor wrote back to me saying, “Don’t presume to tell students that there is ONE CORRECT interpretation of the line.” I responded, “What should I say instead?” She wrote back, “Say, ‘One possible reason is that red roses are traditional symbols of beauty and of romantic love.’” I pointed out that if I were to follow her advice, I would have to include a similar disclaimer (“one possible”) in almost EVERY STATEMENT made about any work of literature in the book, which would make for awkwardness. She informed me that I was being overly directive and not respecting the students’ right to his or her own interpretation and that this made her question my suitability for the job she had asked me to do.

Let me hasten to add that I do understand where that editor was coming from. She held a version of a reader response theory of literature that goes something like this: a text means whatever the reader constructs when reading it. This grotesque misunderstanding of what “a reader’s construction of a text” can reasonably mean had become the de facto orthodoxy in ELA lit texts at the middle-school and secondary-school levels. I call this a grotesque misunderstanding because a text is an act of communication and as such depends, usually, upon shared usages and upon the belief on the part of the reader and the writer that communication across an ontological gap of a communicable meaning is possible. To deny that—to say that any text can mean anything—is to undercut the very notion of communication, of transmission across that gap from one subjectivity to another. Part of teaching people how to read literature is to teach them about conventional usages and what those can reliably be taken to mean.

Now, one might say, but wouldn’t an alternate reading like the following be acceptable?

The convention of the red, red rose as a symbol of feminine beauty is part of an complex of objectifications found in poetry and song produced by men, particularly in the Cavalier and early Romantic periods, and the speaker probably uses this because he is a conventional, unthinking, objectifying pig.

The editor might have had a student response like that in mind.

But here’s the problem with that: the editor would be confusing significance (meaning as mattering to the reader) with interpretation (meaning as the intent of the author). Failure to observe this distinction leads to a lot of complete nonsense in writing and speaking about literary texts. The differing responses are to differing matters–what the author intended and what significance what the author did has for a particular reader.

So, how does all of this relate to the new tests?

Well, one remarkable characteristic of the new tests is that they have COMPLETELY OVERTHROWN what was the STANDARD CHURCH ORTHODOXY in K-12 ELA–the prevailing Reader Response/Constructivist/The Author Is Dead orthodoxy that texts have alternate readings, constructed by readers, that have to be respected. For the most part, the questions about literature on the new exams assume that THERE IS ONE CORRECT ANSWER. Am I the only one to notice that? Did millions of English teachers and textbook writers who were of the “readers construct texts” or “reader response” schools of thought suddenly change their minds about this?

No, their minds were changed for them, de facto, by people constructing the new tests based upon the new standards.

Shouldn’t I be pleased about this, given my defense, above, of the “one true” reading of the line from Burns? No, and here’s why: What we mean by “What does this mean?” itself differs depending upon whether we are talking about intent or significance, and intent itself is by no means cut-and-dried, simply there for discovery. Getting at intent involves a great deal of knowledge of matters like literary conventions and genres and techniques and historical periods and the thought and practice and life experience of the author and much, much else. So even if we made the assumption that any question on a standardized test must deal with intent and not with significance, it would still be the case that particular passages would be open to varying interpretations.

And the relevance of extra-textual matters to interpretation raises another issue with regard to the approach to literature instantiated in the new standards. Students and teachers are encouraged in these to follow a New Critical procedure—to examine closely the text itself, without reference to external materials. But intent does not exist in a vacuum. If someone leaves a note saying, “Tie up the loose ends,” it matters whether the note is from a macramé instructor or from a mob boss worried about possible informants! Texts exist in context.

When I examine the new tests and the questions asked on them, my overwhelming impression is that the questions were written by people who hadn’t the subtlety to understand what a complex business learning to read carefully and well is. As often as not, the questions FAIL because the question writer did not himself or herself understand some subtlety. Let me give an example to illustrate what I mean.

Suppose that a question on one of these tests reads as follows:

Which of the following best describes the attitude of the speaker in the first line of Burns’s “A Red, Red Rose”?

A. Ardent affection
B. Casual interest

The test writer would probably think that answer A. is the correct answer.

But consider this: A deconstruction of that first line would look beyond the verbal intention—the intended communication—to other factors getting at significance. Why did the speaker use a hackneyed, conventional expression? Why did he express the conventional association in a simile instead of in some more sophisticated way? Do the facts that he chose a hackneyed convention and chose the simile, most likely, simply as an easy way to fill out the meter suggest that he did not give this poem much work or thought? In other words, is this first line suggestive of someone who is not as serious as would be another poet who, in this circumstance, would bother to say something original and real? I’m reminded of a fellow I knew when I was a kid who had written what he called a “general purpose love song.” He said to me, “The beauty of this song is that I can throw any girl’s name in there. Miranda. Amanda. Sweet, sweet Jane.” Is the casualness of Burns’s line related to the fact that in the last stanza, he’s outta here?

“Hey, you’re great. Really. I’ll be thinking about ya. Outta here.”

Hmm. Suddenly the wrong answer starts to look as though it might not be so wrong after all because now we are talking not about intent but about significance. Is this an accurate reading of the significance? I love Robbie Burns. I have participated with great delight in Burns dinners (though I shall always pass on the haggis). But he was a notorious womanizer, and this poem is a piece of tossed-off minstrelsy and not a great work of art like his “Song Composed in August” or “To a Mouse.” I don’t mean to take away from the poem by saying that. It’s a perfect specimen of its type. But it’s a conventional type. It’s a “My Darlin’ Clementine,” not Yeats’s “The Folly of Being Comforted” or Millay’s “Love Is Not All” or Burns’s own “John Anderson, My Jo.”

Alfred North Whitehead on Standardized Testing, from his collection of essays, “The Aims of Education” (1929).

Give a copy to your favorite reformer.

Whitehead writes:

“In education, as elsewhere, the broad primrose path leads to a nasty place. This evil path is represented by a book or a set of lectures which will practically enable the student to learn by heart all the questions likely to be asked at the next external examination. And I may say in passing that no educational system is possible unless every question directly asked of a pupil at any examination is either framed or modified by the actual teacher of that pupil in that subject.”

“A common external examination system is fatal to education. The process of exhibiting the applications of knowledge must, for its success, essentially depend on the character of the pupils and the genius of the teacher.”

“The best procedure will depend on several factors, none of which can be neglected, namely, the genius of the teacher, the intellectual type of the pupils, their prospects in life, the opportunities offered by the immediate surroundings of the school and allied factors of this sort. It is for this reason that the uniform external examination is so deadly. We do not denounce it because we are cranks, and like denouncing established things. We are not so childish. Also, of course, such examinations have their use in testing slackness. Our reason of dislike is very definite and very practical. It kills the best part of culture.”