Archives for category: Data
« Older Entries
Newer Entries »

Why You Must Not Be “Data Driven”

By dianeravitch
August 28, 2017 //
70

Reader John Wund left this. Moment about the power of data to mislead those of blind faith:

“Once, after I had an MS in physics, I was hired by a medical school to work on building a ‘physical biochemistry lab’. I also have a degree in Astronomy, so I think I know something (though, perhaps not everything) about statistics.

“Any useful statistic needs to be accompanied by an ‘error estimate’ and needs to be measuring something clearly agreed upon by everyone (brightness, position, etc.). When psychology pretends to measure ‘intelligence’ every rule is broken, and it sends Gauss whirling in his grave.

“These new tests that pretend to measure ‘teaching’ or ‘skill’ or ‘learning’ are simply an extension of the attempt to (or pretense to) measure ‘intelligence’. There is no clear (non-circular) definition of the trait to be measured, or of it’s importance to a larger society. And, no indication of uncertainty (statistical, rather easy to calculate for those who ‘like numbers’) is ever attached.

“It is not statistics, but their bogus use that is the problem. Most people think that numbers and computers are ‘scientific’, and so they must be honored. I have a funny story about that.

“When I went to the Biochemistry Dept., I found that there was only only one other person who had even a passing acquaintance with computers (this was in the late 60’s). Even that person had little understanding of statistical analysis. At first, I took information, analyzed using a calculator it and wrote out the result for the various profs and post-docs. I found they often questioned the result, so I wrote simple programs (Basic or Fortran) and ran them on a computer hooked up to a teletype machine. Suddenly, the questions stopped. After all, if a machine typed it out, it must be factual! Never mind that I was the one programming the machine.

“People are attracted to the certainty that numbers seem to provide. Unfortunately, those numbers can be used to control people, even if they have no value. The ancient Greeks understood that inductive logic always had more value than deductive. We have forgotten that lesson.

“But, it goes even deeper. Because people venerate numbers (and certainty), they are easily manipulated by those who spout them in a blatantly (to me) inaccurate way. Sadly, numbers have become a way to buffalo people. Remember ‘Ivory Soap’, 99 and 44 one hundredths percent pure? Pure what? Pure bullshit.

“Statistics teaches that there is no certainty (well before Heisenberg). There is only a ‘best guess’ at any particular time, subject to change. Statistics revolving around a poorly defined concept are worthless. As a saying in my past explains, GIGO (garbage in, garbage out). Computers and statistics aren’t the problem, our blind belief in the stuff that comes out of machines is.”

Categories Accountability, Data

Emily Talmage: How Data is Destroying Our Schools–and Education

By dianeravitch
August 25, 2017 //
61

Emily Talmage teaches in Maine. She recounts in this post her first encounters with data-driven instruction, both in public schools and in a no-excuses charter school. She has come to understand that the data are not meaningful and that they are inherently at odds with the individual nature of the child, who cannot be defined by numbers. She suspects that someone will use whatever data her classes produces to make money. Better to look at the children in front of you, not the data that reduces them to digits on a spreadsheet.

A few weeks into my first year as a teacher, my colleagues and I met for our first “data team” meeting of the year.

Our principal had printed results from the previous year’s standardized tests and given a copy to each of us.

“Take a few minutes to look at the data, and then we’ll decide what inferences we can make from it,” he instructed.

He had a book with him – something with “data coaches” in the title – and was following a protocol laid out within.

I looked at the graphs, then – smiling – at my principal.

Surely he was joking.

At that point in the year, I had only five students – four third graders and one fifth grader – in a self-contained special ed classroom for kids with severe emotional disturbances. They were children who had experienced extreme trauma and abuse, and who struggled to get through a day at school without an attack of panic, rage, or violence.

All five had gotten one’s – the lowest possible score – on the previous year’s math and reading tests.

“Ms. Kennedy,” our principal said flatly, “what inferences can you make from this data? This is how we will be planning our instruction for the year.”

…

It was my first time experiencing the absurdity of data-driven education, but far from my last.

Then she worked at a no-excuses charter school in Brooklyn, and remembering it still gives her nightmares.

Now teaching in Maine, she is swimming–or drowning–in data, and she learns nothing from it that she didn’t know already.

Yet somewhere, she assumes, someone is figuring out how to monetize the data, even though it is utterly meaningless.

Categories Data

Gary Rubinstein: The Alum-Lie Spreads About College Completion Rates for Charter Students

By dianeravitch
August 10, 2017 //
60

Gary Rubinstein has been bothered by the lies spread about the college completion rates of charter students. He has pointed out that it is unfair to say that X% of your students in twelfth grade finished college without admitting that twelfth grade is not the right place to begin, since it excludes the attrition that may have occurred earlier.

Among charters, KIPP has been honest in stating that it counts the students who completed eighth grade and persisted to high school graduation. Others say they just can’t find the data to learn when their senior started in their school. Can you believe that?

Gary posts a twitter exchange he had with Richard Whitmire, who clearly did not want to engage with Gary. Whitmire wrote a fawning bio of Michelle Rhee, then a book praising Rocketship Charters. Not a high success rate.

What Gary is looking for is candor, not boasting.

It seems the public is beginning to understand that charter school boasting is built on cherrypicking students and pushing out the ones that get low scores.

Categories Accountability, Charter Schools, Data, Education Industry, Higher Education, KIPP Charter Schools, Privatization

Peter Greene: What William Sanders Learned About Teachers by Studying Radioactive Cows

By dianeravitch
May 22, 2017 //
31

We now know that William Sanders, the creator of Value-Added Measurement, which measures teacher quality with the methods of an agricultural statistician, is revered in the think tank world of D.C.

But Peter Greene, who teaches in a Pennsylvania high school, doesn’t think much of VAM.

He recounts how Sanders was inspired to think about measuring teachers based on his studies of radioactivity in cows that were downwind from a nuclear explosion (really!).

To which Sanders writes:

Oh, let’s tell the truth. VAM systems have also been limited by the fact that they’re junk, taking bad data from test scores, massaging them through an opaque and improbable mathematical model to arrive at conclusions that are volatile and inconsistent and which a myriad educators have looked at and responded, “Well, this can’t possible be right.”

You’ll never find me arguing against any accountability; taxpayers (and I am one) have the right to know how their money is spent. But Sander’s work ultimately wasted a lot of time and money and produced a system about as effective as checking toad warts under a full moon– worse, because it looked all number and sciencey and so lots of suckers believed in it. Carey can be the apologist crafting it all into a charming and earnest tale, but the bottom line is that VAM has done plenty of damage, and we’d all be better off if Sanders had stuck to his radioactive cows.

Categories Accountability, Data, Hoax, Teacher Evaluations, Testing, VAM

Chicago: Parent Discovers Massive Breach of Privacy of Student Records

By dianeravitch
March 11, 2017 //
10

A parent in Chicago discovered a massive breach of private data about students in private schools receiving special education services. The data was controlled by Chicago Public Schools, but obviously with little regard for privacy. The parent was a student Privacy activist, Cassie Creswell.

The following post is by Cassie Creswell, a Chicago parent activist from Raise Your Hand Illinois and a key member of our Parent Coalition for Student Privacy. In January, Cassie also testified on our behalf at the Chicago hearings of the Commission for Evidence-Based Policy against overturning the ban to enable the federal government to create a comprehensive student database of personally identifiable information.

More recently, upon examining expenditure files on the Chicago Public School website, Cassie discovered the names of hundreds of students along with the disability services they received at numerous private and parochial schools. She immediately contacted several reporters, and though an article in the Sun-Times subsequently briefly reported on this breach, the reporter did not mention that it was primarily private and parochial students whose data was exposed. In addition, legal claims for special education services that CPS had originally rejected were included along with student names. Cassie’s fuller explanation of this troubling violation of student privacy is below — as well as the fact that at least some of these schools and families have still not been alerted to the breach by CPS.

Cassie writes:

Once again, Chicago Public Schools has improperly shared sensitive student data, the Chicago Sun-Times reported on February 25th.

Medical data about students used to administer outsourced nursing services was stored on an unsecured Google doc available to anyone with the link. And personally-identifiable information (PII) about students with Individualized Education Programs (IEPs), including their name, student identification numbers and information about services and diagnoses related to their disabilities, were included in files of detailed vendor payments posted on the district’s public website.

I discovered this latter information in the vendor payment data, while in the course of searching for information about standardized testing expenditures. The files covered seven fiscal years, 2011-2016, but were only posted on the CPS website this past summer. Noticing what appeared to be a student name and ID number listed in the file struck me as surprising and likely a privacy violation. All in all, there were more than 4500 instances in the files where students’ names appeared along with the special education services they received.

Upon closer examination, it was clear to me that there was a great deal of highly sensitive student personal information that had been disclosed, with payments made from CPS to educational service providers assigned to hundreds of students with special needs attending private schools as well as public schools. Included were the name of the students, the schools in which they were enrolled, their ID numbers, the vendors who had been hired and the services they provided according to the students’ diagnoses. The funds for the payments came from public funds routed through the students’ home districts, CPS, to fulfill requirements of the federal Individuals with Disabilities Education Act (IDEA) for spending on special education students enrolled in private schools.

This breach has since been confirmed as violating federal and state privacy laws — at least in the case of the public school students whose personal information was disclosed and likely the private school students as well.

Categories Data, Privacy, Special Education, Students

Gene V. Glass: The Department of Education’s “Gold Standard” Produces “Fool’s Gold”

By dianeravitch
February 21, 2017 //
13

Gene V. Glass is one of our nation’s most distinguished education researchers.

This post is an important analysis of the failure of the U.S. Department of Education’s Institute of Education Sciences, which was reorganized during the George W. Bush administration.

As a new administration moves into the US Department of Education, the opportunity arises to review and assess the Department’s past practices. A recent publication goes to the heart of how US DOE has been attempting to influence public education. Unfortunately, in an effort to justify millions of dollars spent on research and development, bureaucrats pushed a favorite instructional program that teachers flatly rejected.

The Gold Standard

There is a widespread belief that the best way to improve education is to get practitioners to adopt practices that “scientific” methods have proven to be effective. These increasingly sophisticated methods are required by top research journals and for federal government improvement initiatives such as Investing in Innovation (i3) Initiative to fund further research or dissemination efforts. The US DOE established the What Works Clearinghouse (WWC) to identify the scientific gold-standards and apply them to certify for practitioners which programs “work.” The Fed’s “gold standard” is the Randomized Comparative Trial (RCT). In addition, there have been periodic implementations of US DOE policies that require practitioners to use government funds only for practices that the US DOE has certified to be effective.

However, an important new article published by Education Policy Analysis Archives, concludes that these gold-standard methods misrepresent the actual effectiveness of interventions and thereby mislead practitioners by advocating or requiring their use. The article is entitled “The Failure of the U.S. Education Research Establishment to Identify Effective Practices: Beware Effective Practices Policies.”

The Fool’s Gold

Earlier published work by the author, Professor Stanley Pogrow of San Francisco State University, found that the most research validated program, Success for All, was not actually effective. Quite the contrary! Pogrow goes further and analyzes why these gold-standard methods can not be relied on to guide educators to more effective practice.

Researchers have told us that we need “randomized comparative trials” to reach “research-based conclusions.”

In fact, says Glass and the article he cites, this is not what happens. And the results of these trials turn out to be easily manipulated and falsified.

He writes:

Key problems with the Randomized Comparative Trial include (1) the RCT almost never tells you how the experimental students actually performed, (2) that the difference between groups that researchers use to consider a program to be effective is typically so small that it is “difficult to detect” in the real world, and (3) statistically manipulating the data to the point that the numbers that are being compared are mathematical abstractions that have no real world meaning—and then trying to make them intelligible with hypothetical extrapolations such the difference favoring the experimental students is the equivalent of increasing results from the 50th to the 58th percentile, or an additional month of learning. The problem is that we do not know if the experimental students actually scored at the 58th or 28th percentile. So in the end, we end up not knowing how students in the intervention actually performed, and any benefits that are found are highly exaggerated.

The sad part of the story is that we now have a new administration that is both ignorant of research and indifferent to it. DeVos has seen the failure of school choice in her own state, which has plummeted in the national rankings since 2003, and it has had no impact on her ideology. Ideology is not subject to testing or research. It is a deep-seated belief system that cannot be dislodged by evidence.

Categories Betsy DeVos, Data, Research, U.S. Department of Education

Reader: Why the $3.5 Billion School Improvement Grants Failed

By dianeravitch
January 19, 2017 //
11

This morning, I posted an evaluation by Mathematica Policy Research, which concluded that the federal School Improvement Grants had no effect on test scores. A reader named Sara explains here why the SIG program failed, after spending $3.5 billion:

 

 

The SIG required certain interventions and did not give any autonomy or decision making power to the people who already worked in the school.

 

So for example in the school where I work, SIG required that an outside organization provide social emotional support to students- rather than supplementing the counseling and social work staff who are highly qualified and already know the students. Whenever new people come into a situation there is a long learning curve. Also people from an outside organization do not have a long term commitment to the school.

 
Another example, staff came in for the grant who merely measured and “coached” – what the school really needed was smaller class size, so for example another math teacher instead of a “coach.” Experienced teachers for the most part know what to do, they are just overwhelmed by the large number of students who have special issues – and they do not have support.

 
Hundreds of thousands of dollars were spent on technology – but the librarian and IT person were let go.

 
The presumption on the part of the administrators (not in the school) of the grant was that the problems in the school lay with the teachers – not with poverty, an insufficient number of qualified staff, and an unstable district.

Categories Accountability, Closing schools, Data, Duncan, Arne, Failure

Stephen Henderson: Betsy DeVos Twists Facts to Advance Her Privatization Agenda

By dianeravitch
January 5, 2017 //
7

Stephen Henderson is the editorial page editor of the Detroit Free Press. He is not anti-charter; his own children attend a Detroit charter school. He is opposed to lies and propaganda. He has written that the charter movement has done nothing to lift the children of Detroit, and that there are as many bad charter schools as public schools. He has written critically of DeVos’s successful efforts to torpedo accountability and oversight of charter schools.

 

When it comes to data and research, he says, DeVos is not to be trusted.

 

He writes:

 

A true advocate for children would look at the statistics for charter versus traditional public schools in Michigan and suggest taking a pause, to see what’s working, what’s not, and how we might alter the course.

 

Instead, DeVos and her family have spent millions advocating for the state’s cap on charter schools to be lifted, so more operators can open and, if they choose, profit from more charters.

 

Someone focused on outcomes for Detroit students might have looked at the data and suggested better oversight and accountability.

 

But just this year, DeVos and her family heavily pressured lawmakers to dump a bipartisan-supported oversight commission for all schools in the city, and then showered the GOP majority who complied with more than $1 million dollars in campaign contributions.

 

The Department of Education needs a secretary who values data and research, and respects the relationship between outcomes and policy imperatives.

 

Nothing in Betsy DeVos’ history of lobbying to shield the charter industry from greater accountability suggests she understands that.

 

If she’s confirmed, it will be a dark day for the value of data and truth in education policy.

Categories Accountability, Betsy DeVos, Charter Schools, Corporate Reformers, Data, Detroit, Education Industry, Michigan, Privatization, Research

Bruce Baker: Why Do States Keep Using Stupid Measures to Evaluate Teachers and Schools?

By dianeravitch
January 4, 2017 //
73

 

 

Bruce Baker of Rutgers University is frustrated. He and colleagues have published study after study about the uses and misuses of standardized test scores to measure teachers and schools.The evidence is clear, he writes. Yet states remain devoted to failed, erroneous methods that pack any evidence!

 

“It blows my mind, however, that states and local school districts continue to use the most absurdly inappropriate measures to determine which schools stay open, or close, and as a result which school employees are targeted for dismissal/replacement or at the very least disruption and displacement. Policymakers continue to use measures, indicators, matrices, and other total bu!!$#!+ distortions of measures they don’t comprehend, to disproportionately disrupt the schools and lives of low income and minority children, and the disproportionately minority teachers who serve those children. THIS HAS TO STOP!”

 

 

Categories Accountability, Data, Failure, Standardized Testing, Teacher Evaluations, VAM (value-added modeling)

Pasi Sahlberg on the Beauty of Small Data

By dianeravitch
December 12, 2016 //
24

Pasi Sahlberg is the great Finnish educator whose book Finnish Lessons gave us a vision of a nation that succeeds without high-stakes testing, without standardized testing, and without charter schools or vouchers. He wrote of highly educated teachers who have wide scope and autonomy in their classrooms and who collaborate with their colleagues to do what is best for their children. He wrote of a national school system that values the arts, physical activity, and play. And, lo and behold, the OECD calls it the best school system in the world!

 

So entranced was I but what I read about Finland that I visited there a few years ago and had Pasi as my guide. The schools and classes were everything he claimed and more.

 

Pasi, like many other education experts, is aghast at the GERM (Global Education Reform Movement) that has swept the world. The agency that has spread GERM far and wide is international testing, the great horse race that only a few can win. Since most are losers, the frenzy for more testing becomes even stronger.

 

Pasi suggests a different approach. Instead of Big Data, produced by mass standardized testing, why not search for small data? 

 

Here is Pasi’s thumbnail sketch of the contrast between Big Data and small data:

 

Big data is a commonly used term in daily discourse that often comes with a label that big data will transform the way we think, work, and live. For many of us, this is an optimistic promise, while for others it creates anxiety and concern regarding control and privacy. In general terms, big data means data of very large size to the extent that its manipulation and management present significant practical challenges.
The main difference between big and small data in education is, of course, the size of data and how these data are collected and used. Big data in education always requires dedicated devices for collecting massive amounts of noisy data, such as specific hardware and software to capture students’ facial expressions, movements in class, eye movements while on task, body postures, classroom talk, and interaction with others. Small data relies primarily on observations and recordings made by human beings. In education, these include students’ self-assessments, teachers’ participatory notes on learning process, external school surveys, and observations made of teaching and learning situations.

 

To watch and listen to Pasi, introduced by Howard Gardner, here is the lecture he gave at Wellesley College in October. 

Categories Data, Education Reform, International, Real Education, Teachers and Teaching
« Older Entries
Newer Entries »