Julia Sass Rubin here analyzes “school report cards” in New Jersey.
This analysis was published last year but is as valid now as it was last May.
Rubin writes:
“Comparing schools to those with similar demographics is a good idea that highlights that students’ personal characteristics play a bigger role in determining their academic outcomes than anything that happens to them in-school. And what could be bad about giving parents and educators more information?
“Unfortunately, rather than providing useful data, the new reports undercut New Jersey’s excellent public schools. The reports also create incentives for districts to manage to the new standards through policies that produce higher rankings but may not meet students’ needs.
“There are four types of problems with the new school reports: artificially created competition, poorly designed comparison groups, arbitrary category definitions, and inaccurate data.”
She then describes the errors inherent in each of these measures. They are as misleading as the A-F report cards, which are often based on the same metrics.
I don’t know about you, but I’m getting sick of the unending efforts to find a scale that can quantify children, teachers, schools. I much prefer a complex qualitative report that helps me understands needs and strengths and that leads to improvement, not punishments.
Surly all those statisticians can find something useful to do in industry or agriculture or public health. Big Data has its limits. The more we learn about how it distorts values and degrades education as it should be, the less it is needed to measure children and the quality of learning.
Diane Ravitch's blog 
This report really shows how data can obscure what is really going on in a school. The release of this data without an analysis such as Rubin’s attached provides no information of value.
LikeLike
Arne Duncan, Bill Gates, Michele Rhee seem to have not learned the Heisenberg Uncertainty Principle. They also do not appear to have learned that controlled experiments can only measure one factor at a time. When multiple factors are involved, it is junk sciene to accept that a standardized test score is only a measure of teacher effectiveness.
LikeLike
I’m sure regulars to this Blog remember the brouhaha over IQ tests—in fact it flares up from time to time.
IQ tests are based on a theory that we can give someone a test to measure cognitive ability.
But, even if the IQ theory is right, IQ tests do not measure a child’s motivation, discipline level, home environment, ability to set goals, if the child has learning disabilities, if the child is eating a regular nutritious brain food diet, if the child prefers water over sodas, if the child has abusive parents, lives in poverty, if the child reads at home, if the parents even speak English, if the child lives in a barrio dominated by violent street gangs, if the child does homework, if the child pays attention in class, etc.
Standardized tests are no better than an IQ test to discover who is going to find a job and be successful at life by earning enough money to live a middle class lifestyle.
The logic behind judging teachers through a standardized test is just as faulty. For instance, the standardized tests do not measure the quality of lesson a teacher teaches and the test doesn’t know the children who paid attention or not.
And if anyone needs proof, here’s my story. I was born into poverty and I had a severe learning disability. When I was seven, I was tested at school and my mother was told I was too retarded to learn to read and write. Well, we can see that was wrong.
I was also a horrible student in class. What I learned or didn’t learn in the public schools had nothing to do with the quality of teaching.
Both of my parents were high school dropouts, not because they wanted to drop out but because the Great Depression forced them to drop out at 14 to survive or starve. My father was a chain smoking, gambling alcoholic who often cheated on my mother even with her best friend, my godmother. My older brother was twelve when I was born and by seventeen, he was already in prison for his first term. My brother, as a teen, ran with a street gang, he was an alcoholic, he chain smoked; he grew up illiterate and died that way at 64. He also often cheated on his wives. My brother had seven children with those two wives. Only two of those seven children finished high school and can read but his oldest daughter harbored a fugitive, her boyfriend, who was on the FBI’s top ten most wanted list and she ended up going to prison. His oldest daughter was also sexually abused at the age of six by her step-father after my brother divorced her mother. The last time I saw my brother’s family was shortly before he died. All of his children were adults and all but two lived in poverty and were illiterate just like their father.
When I was in school K – 12, I never did homework and seldom paid attention in class. My mother–with the helpful advice from my first grade teacher—defied the test that said I would never learn to read and write taught me to read at home using a wire coat hanger as a painful motivator. She lived with guilt up to her death at 89 and begged my forgiveness on her death bed for beating me with that coat hanger to force me to learn to read. I told her there was nothing to forgive because if she hadn’t done that, I would have followed my brother’s lifestyle into gangs, drugs, alcohol and poverty.
In conclusion, the public school test culture being driven by the robber barons and wolves of Sesame Street supported by the traditional media, the White House, the Congress is a con-game designed to mislead an ignorant public. It is a logical fallacy to claim that a standardized bubble test given to a child at any given moment will reveal incompetent teachers and failing schools.
If I was seven now with the same parents and home environment and took one of those tests today, it would not be fair to judge the teacher by the results of the test I took, because I would have done horrible on that test at any age.
My teachers might have taught incredible lessons for everything on those tests but it would do no good when I was zoned out for a variety of reasons that had nothing to do with the public schools and wasn’t paying any attention. And the teacher, by law, cannot use a club or paddle to force me through the threat of pain to pay attention, read the assignments, and do the classwork and homework. That of course brings me to another thought: What if the world these robber barons and wolves really want is a world of legal torture in a privatized education system to force very kid to learn or face prison and death.
What saved me from poverty was a desperate mother who saw her older illiterate son running with street gangs as a teenager, cutting school, using drugs, drinking too much booze and going to prison for fifteen years of his 64 on this planet. She didn’t want that for me and after she whipped the tears from her eyes and face, she went to my first grade teacher and asked her for help—what can I do to save my youngest son. The rest is history. My mother was one of the few parents from that world who actually asked one of my public school teachers for help and then followed that advice to the letter. Few parents from that world do that. I know of one from the thirty years I taught kids who came from the same world I was born into. Of course my painful path to literacy didn’t end there. I barely graduated from high school and had no intention of going to college so I joined the Marines and out of boot camp was sent to Vietnam. A few years later, an older more mature young man decided that maybe college wasn’t such a bad idea because he didn’t want to spend the rest of his life washing dishes in a restaurant working long hours for poverty wages. Instead, he changed direction and went to college on the GI Bill—and those first two years were hard for me because I had to learn how to be a student on my own.
No standardized test would have been able to predict the path I walked as a child into the adult world and no standardized test can measure the quality of a teacher or the teacher’s lessons. If any standardized test today had been used back then to judge my teachers, I wonder how many would have lost their jobs and how many of the schools I attended would have been branded failing schools because of the world of poverty I was born into.
For that reason, I never used tests in the classes I taught to judge a student’s learning or ability to learn. The grade earned in the classes I taught came from the work the students turned in, and the results of that work as I graded it often dictated if I had to reteach a lesson using different strategies to get a concept across for the students who were paying attention and doing the work.
And of course there was the never ending struggle to reach the kids who were like me at the same age—the kids who did not read, do the homeowner, cooperate in class, do the classwork, pay attention, etc.
LikeLike
Lloyd, I tweeted this blog entry and also posted it on Facebook calling attention to your comment. Every teacher and health and human services professional who works at a school as well as all administrators should read your entry. These measurements by which we are being scrutinized are evil wrapped in a business suit and will destroy all schools being evaluated with them as well as all human inhabitants contained in them.
LikeLike
I was so fortunate that both of my parents who were both forced to drop out of high school (at age 14) to survive during the Great Depression because they were both literate and loved to read—even more than watching TV.
My father, who was a chain smoking gambler and alcoholic, haunted used bookstores (books were cheaper) and read paperback westerns and mysteries. My mother read sanitized romances and the Bible (over and over). If a romance had explicit sex, she burned the book. I grew up reading in the living room while the TV was on. My dad and mom were reading their books at the same time while a cloud of cigarette smoke coated the ceiling in a gray haze.
That’s the home environment I grew up in. For my first decade, my parents lived in poverty and then started the slow climb out into the lower rungs of the middle class where we stayed.
What about all of the homes where the parents are illiterate or functionally illiterate and there are no books, magazines or newspapers around because the parents have no love of literature of any kind? Can we really expect those children to come to school and fall in love with reading just because someone in Washington DC, who is financially supported by a billionaire plutocrat or two, says so through laws passed by Congress and signed by the President.
Government can’t legislate a passion for reading.
Government can’t legislate more than 4 million super teachers as the flawed “Waiting for Superman” implies.
The best we can ask for is more than 95% of teachers in the US are dedicated and willing to work harder teaching America’s children than most Americans will ever work in other professions.
There is no magic pill/bullet that will fix all the challenges that walk into classrooms across America every school day.
These billionaire plutocrats and the federal government must but out of education and support teachers so they have the tools to do what they must do: teach!
If they don,t then the United States is headed down a path that will lead to a brutal, Fascist dictatorship and that’s money you can take to the bank except the plutocrats will own the banks too.
Anyone who reads and studies history should know this because it has happened before. When a few powerful men think they have the answers that will solve all the problems in a country, when they fail, which they will, then they crack down to force their vision on people through tyranny. Instead of admitting they are wrong, these plutocrats will blame the people and punish them, and if even one or two in their own ranks stand up and say it is wrong, the group will punish those few who dared to have the courage to speak out.
For instance, let’s say Bloomberg (selected at random for no reason) ten years from now publicly says the other billionaire plutocrats are wrong and he was wrong and what they are doing to the country is wrong. The pack will turn on Bloomberg and tear him apart. He will be branded through the media as a traitor and publicly destroyed. His wealth and power will not be enough to stand up to their combined wealth and power.
LikeLike
“Surly all those statisticians can find something useful to do in industry or agriculture or public health. Big Data has its limits. The more we learn about how it distorts values and degrades education as it should be, the less it is needed to measure children and the quality of learning.”
I had used this quote as a response to an earlier post, but it is relevant here again. It is from Jaron Lanier’s “You Are Not A Gadget”:
“Information systems need to have information in order to run, but information under represents reality.”
LikeLike
From NJ DOE: Peer Schools are schools that have similar grade levels and students with similar demographic characteristics, such as the percentage of students qualifying for Free/Reduced Lunch, Limited English Proficiency programs or Special Education
programs.
You will find school A in a “peer” group with school B, YET school B may NOT have school A in its “peer” group! Does that make any sense? In “peer” groups, FRPL can vary by 20% and SpED by 13%.
If you read through these “progress reports” it is easy to spot errors. It makes one wonder if the actual scores of % passing are correct!
LikeLike
Think of the figures generated by IQ tests, high-stakes standardized tests, VAM, etc., as descriptive or summative.
They collapse lots of information into one number.
That’s their greatest strength.
That’s also their greatest weakness.
When they are used without thought to context, with suspension of one’s judgment and good sense, with little or no awareness of how they are generated and arrived at, without even an inkling of what they do measure [very little] and what they don’t measure [a great deal]—
You have a exemplar of how mathematical intimidation and obfuscation is used by the self-styled “education reformers” as a mighty club to beat down and degrade OTHER PEOPLE’S CHILDREN and the schools they attend. The vast majority.
Where is this mighty club of data-driven instruction and management used very very sparingly, if at all? When it comes to THEIR OWN CHILDREN and the schools they attend. The advantaged few.
Any wonder why I call them edubullies and their unethical numbers/stats people accountabully underlings?
😎
LikeLike
Random irrelevant comment alert: Julia Sass Rubin has leaped onto my list of top 3 favorite education-related names, trailing only Hanna Skandera and my runaway #1, Payroll and Procurement Liaison Lynn Pomerantz Antwork.
LikeLike
¡Qué!
LikeLike