Archives for category: Data and Data Mining
« Older Entries
Newer Entries »

How Statistics Can Be Manipulated Any Which Way

By dianeravitch
July 8, 2014 //
43

Mr. Anonymous, an education policy analyst who is working towards his doctorate, wrote the following cautionary story about the use and misuse of statistics for political purposes. He requires anonymity for the usual reasons, mostly fear of retaliation for speaking up.

He writes:

The Common Core and Departments of Education: Lies, Darn Lies, Statistics and Education Statistics

Numbers have taken center stage in the discussion of education policy in the United States. Test score metrics have become a particularly critical set of numbers. They are seen as objective measuring devices, comparable across years, that provide a reliable evaluation of how students, teachers, schools, districts, and the United States as whole are doing. But are they really objective?
The push for implementation of Common Core exams has caught the attention of the public. In New York State, as in many other states across the nation, questions have been raised about the motivations of those pushing for the roll-out of these exams and their use in high-stakes evaluations. As we will see below such concerns are definitely legitimate given the history of the New York State Department of Education and the Board of Regents in setting cut-scores and changing exams in ways that serve political and other ends.

Let’s start with Biology, a standard course that almost every high school freshman takes. Remember dissecting that frog? In 2001 the New York State Department of Education changed the Biology Regents to a re-named “Living Environment.” A rather remarkable aspect of the change was the dramatic lowering of the passing score. In the Biology exam a student needed at least 59 points (out of a total of 85 possible points) to earn a passing grade of 65. On the new Living Environment Regents students need only 40 points (out of a total of 85 possible points) to earn a passing grade of 65. In some years (e.g. 2004) a student needed only 38 out of 85 points to earn a passing grade of 65.

The story repeats itself in mathematics. Until 2002 the New York State Department of Education required students to take a “Sequential Mathematics I” exam. That test had a total point value of 100 points. The conversion was simple enough, each point was equal to one point and a student needed 65 points to pass. Then, in 2002, the math exam was switched to a “Mathematics A” exam.

On this test students needed to score 35 out of a possible 84 points to earn a 65 and pass. Earning 42% of the possible points led to a 65. Then, in 2008, the math exam was switched again, this time to an “Integrated Algebra” exam. On this test students needed to earn 30 out of a possible 87 points to earn a 65 and pass. Earning 34% of the possible points now led to a 65.
The United States and Global History exams underwent similar changes at the turn of the millennium. Before the changes students were required to write 3 essays accounting for 45% of their final score. After the changes students were required to write only two essays accounting for only 35% of their final score. On one of the essays students are provided with extensive information they can use in their writing.

A couple of years later the exact same process occurred with the English Regents. In 2011 the New York State Department of Education changed the exam from a two part six hour test with two essays to a single part three hour test with only one essay. Again the cut scores were dramatically lowered. The scales on these two exams are very different making comparison difficult. One way to measure the change is to look at the grade a student would receive if s/he got exactly half the multiple choice questions correct and earned exactly half of the possible points on the essay(s). On the old English exam that student would have received a grade of 43. On the new English exam a grade of 50.

A year ago the New York State Department of Education changed things yet again. But this time they did not change the exam. They just changed the cut scores. From 2011 until 2013 out of 286 possible point combinations on the exam an average of 74 resulted in a passing grade. Then, in June of 2013, the number of point combinations leading to a passing grade was dramatically lowered by 23%. Since then an average of 63 point combinations out of 286 leads to a passing grade.

It is disturbing that this change occurred at the very moment when the test results would first be used to evaluate teachers. The research base shows that such value-added metrics are unreliable. For example a RAND report concluded “the research base is currently insufficient for us to recommend the use of VAM for high-stakes decisions.” A report out of Brown University concluded “the promise that value-added systems can provide such a precise, meaningful, and comprehensive picture is not supported by the data.” Nonetheless New York State passed laws requiring school districts to use test scores in teacher evaluations. Why, at the same time, did the Department of Education quietly change the cut scores on the English Regents? Is it an attempt to ensure that more teachers are rated ineffective? This would allow certain interest groups to declare the law a success and claim that “bad teachers” are now being identified and should be fired. Is it an attempt to create evidence that there is an epidemic of failing students in New York State? This would allow certain interest groups to proclaim that the crisis can only be solved if the new Common Core Standards are implemented without delay.

Advocates of the Common Core are either ignorant of or deliberately ignore this history. A decade ago New York State Department of Education decided that the high school graduation rate was too low. They therefore changed exams and cut scores to make them easier. The graduation rate went up. Now it seems that some powerful interests have decided that it is too easy to graduate. So they want the exams made harder and the passing cut scores raised. It is evident from the history reviewed above that playing with cut scores is not the way to improve education. After all that just leaves us in the very place we are in today. Yet we seem to be condemned to repeat this cycle all over again. We seem to be enamored of easy solutions. Make exams harder (or easier). Raise cut scores (or lower them). What we do not seem to be willing to do as a nation is roll up our sleeves and do the really, really hard work of ensuring that every student receives a quality education.

Categories Data and Data Mining, Scandals Fraud and Hoaxes, Teacher Evaluations, Testing

Jason France: The Data Crises in Louisiana

By dianeravitch
July 7, 2014 //
3

Jason France (aka blogger Crazy Crawfish) writes here about the warping and destruction of data held by the Louisiana Department of Education.

He writes:

“There is a data crisis at LDOE. Almost all of the data collection systems are failing. The data, statistics and reports being generated are garbage. Data is being ferried back and forth between the department and school districts using Excel worksheets and through e-mail correspondence. This leaves many students at high risk to data theft and privacy violations. Because the systems impacted are numerous and core to much of the reporting and analysis performed by the Department, it is impossible for LDOE to claim they are reporting accurate or reliable numbers for dropouts, graduates, TOPS scholarship awards, school performance scores, test scores, student counts and breakdowns for MFP funding, program counts. . . the list goes on and on. The situation is really serious and probably just about hopeless at this point.

“I will explain how this situation developed and give specific examples of systems, impacted and correspondence I’ve received from school districts trying to work with the department.

“This crisis was created intentionally by John White and his second in command that he brought with him from New York, Kunjan Narechania. White did not really care what the data said, because he had already determined the outcome for many of his programs. (I don’t think he was also not planning to be here longer than 2 years when all the cut-backs and destruction he’d wrought really started to impact daily operations.) White undertook a slash and burn campaign on the department’s data and analysis folks and immediately implemented policies that guaranteed data would deteriorate immediately. White abandoned a 4 million dollar warehouse named LEDRS we were just finishing. . . as he arrived on the scene, but not before using it to transmit almost all of the data contained in the Warehouse to CREDO to produce reform friendly propaganda masquerading as true data analysis.”

Read on for a remarkable story.

Categories Data and Data Mining, Louisiana

What World Does the NY Times Editorial Board Live in?

By dianeravitch
June 26, 2014 //
106

Yesterday, the New York Times published an editorial vigorously agreeing with the Obama administration’s plan to give ratings to colleges and universities and agreeing with Education Trust that federal aid to colleges should be tied to those ratings. EdTrust was and remains one of the strongest supporters of NO Child Left Behind, having helped to write that abominable law.

On principle, I oppose the ratings game and believe it will turn into NCLB for higher education, with incentives that undermine the mission of the institutions as they get caught up in the numbers game. How will colleges measure the “value-added” of courses in philosophy, ancient history, art, and music?

The Times apparently doesn’t read its own stories. It doesn’t recognize that students are discouraged not by a lack of information but by the crushing debt they incur. Why don’t we have low-cost or free public colleges? The Times has reported in the past about how states have shifted costs from the public to students. Why is this not a more pressing need than data?

When the Times says that the U.S. has among the lowest college graduation rates in the developed world, it should have mentioned that one nation with a much lower rate is Germany, the dominant economy in Europe. What does Germany know that we don’t know?

If the Times thinks that getting a higher college graduation rate matters, why not propose ways to reduce the cost to students? The greatest barrier to college access and completion is affordability, not lack of data.

Categories Data and Data Mining, Higher Education, NCLB (No Child Left Behind)

Stephanie Simon: How Parents Organized and Beat Big Data

By dianeravitch
June 11, 2014 //
8

Stephanie Simon writes in politico.com about how parents organized, lobbied, agitated, and brought down inBloom.

Simon writes:

“You’ve heard of Big Oil and Big Tobacco. Now get ready for Big Parent.

“Moms and dads from across the political spectrum have mobilized into an unexpected political force in recent months to fight the data mining of their children. In a frenzy of activity, they’ve catapulted student privacy — an issue that was barely on anyone’s radar last spring — to prominence in statehouses from New York to Florida to Wyoming.”

Most shocking of all is that the Obama administration is prepared to spend $1 billion (half from the federal government, half from the states) to track the movements of every child:

“Now, parents are rallying against another perceived threat: huge state databases being built to track children for more than two decades, from as early as infancy through the start of their careers.

Promoted by the Obama administration, the databases are being built in nearly every state at a total cost of well over $1 billion. They are intended to store intimate details on tens of millions of children and young adults — identified by name, birth date, address and even, in some cases, Social Security number — to help officials pinpoint the education system’s strengths and weaknesses and craft public policy accordingly.

“The Education Department lists hundreds of questions that it urges states to answer about each child in the public school system: Did she make friends easily as a toddler? Was he disciplined for fighting as a teen? Did he take geometry? Does she suffer from mental illness? Did he go to college? Did he graduate? How much does he earn?

“Every parent I’ve talked to has been horrified,” said Leonie Haimson, a New York mother who is organizing a national Parent Coalition for Student Privacy. “We just don’t want our kids tracked from cradle to grave.”

Why does the Education Department want so much information about every child? What is the rationale for assembling Big Data about our children? Does Congress know about this? Is there any other government in the world that is data mining its children?

Will parents mobilize to stop the federal government from mining their children’s personal data?

Categories Data and Data Mining, Parent Groups, Parents, U.S. Department of Education

Bob Shepherd on the Insanity of “Education Reform”

By dianeravitch
June 1, 2014 //
69

Bob Shepherd writes on the absurd demands now placed on teachers and principals by politicians, who expect to see higher test scores every year. Step back and you realize that the politicians, the policy wonks, the economists, and the ideologues are ruining education, not improving it. They are doing their best to demoralize professional educators. What are they thinking? Are they thinking? Or is it just their love of disruption, let loose on children, families, communities, and educators?

Bon Shepherd writes:

OK, you are sitting in your year-end evaluation session, and you’ve heard from every other teacher in your school that his or her scores were a full level lower this year than last, and so you know that the central office has leaned on the principal to give fewer exemplary ratings even though your school actually doesn’t have a problem with its test scores and people are doing what they did last year but a bit better, of course, because one grows each year as a teacher–one refines what one did before, and one never stops learning.

But you know that this ritual doesn’t have anything, really, to do with improvement. It has to do with everyone, all along the line, covering his or her tushy and playing the game and doing exactly what he or she is told. And, at any rate, everyone knows that the tests are not particularly valid and that’s not really the issue at your school because, the test scores are pretty good because this is a suburban school with affluent parents, and the kids always, year after year, do quite well.

So whether the kids are learning isn’t really the issue. The issue is that by some sort of magic formula, each cohort of kids is supposed to perform better than the last–significantly better–on the tests, though they come into your classes in exactly the same shape they’ve always come into them in because, you know, they are kids and they are just learning and teaching ISN’T magic. It’s a lot of hard work. It’s magical, sometimes, of course, but its’ not magic. There’s no magic formula.

So, the stuff you’ve been told to do in your “trainings” (“Bark. Roll over. Sit. Good Boy”) is pretty transparently teaching-to-the-test because that’s the only way the insane demand that each cohort will be magically superior to the last as measured by these tests can be met, but you feel in your heart of hearts that doing that would be JUST WRONG–it would short-change your students to start teaching InstaWriting-for-the-Test, Grade 5, instead of, say, teaching writing. And despite all the demeaning crap you are subjected to, you still give a damn.

And you sit there and you actually feel sorry for this principal because she, too, is squirming like a fly in treacle in the muck that is Education Deform, and she knows she has fantastic teachers who knock it out of the park year after year, but her life has become a living hell of accountability reports and data chats to the point that she doesn’t have time for anything else anymore (she has said this many times), and now she has to sit there and tell her amazing veteran teachers who have worked so hard all these years and who care so much and give so much and are so learned and caring that they are just satisfactory, and she feels like hell doing this and is wondering when she can retire.

And the fact that you BOTH know this hangs there in the room–the big, ugly, unspoken thing. And the politicians and the plutocrats and the policy wonks at the Thomas B. Fordham Institute and the Secretary of the Department for the Standardization of US Education, formerly the USDE, and the Vichy education guru collaborators with these people barrel ahead, like so many drunks in a car plowing through a crowd of pedestrians.

Categories Accountability, Corporate Reformers, Data and Data Mining, Principals, Teacher Evaluations, Teachers and Teaching

ConnectEDU Goes Bankrupt, 20 Million Student Data Files at Risk

By dianeravitch
May 30, 2014 //
45

Leonie Haimson, leader of Class Size Matters and a student privacy hawk, reports on the bankruptcy of ConnectEDU. Data mining is big business.

She writes that:

“ConnectEDU was one of the three data dashboard companies chosen by the NY State Education Department to receive a statewide set of personal student data through the inBloom data cloud, as part of their “EngageNY Portal.” Now ConnectEDU has announced it has gone bankrupt, despite receiving a $500,000 grant from the Gates Foundation less than a year ago….The assets of ConnectEDU, including 20 million personal student records, are being bought by a venture capital company called North Atlantic Capital. Now the FTC is stepping in, to try to block the handing over of all these personal records. According to Education Week:
The potential sale of 20 million student records by ConnectEDU, an ed-tech company that filed for bankruptcy in April, has prompted the Federal Trade Commission to step in to protect the student data, the agency announced Friday.

“ConnectEDU, a 12-year-old Boston-based company, provides interactive tools to help K-12 and post-secondary learners make academic and career decisions. In its privacy policy, ConnectEDU promised that—prior to any sale of the company—registered users would be notified and have the ability to delete their personally identifiable data.”

“

Categories Data and Data Mining, Privacy and Privacy Rights

Laura Chapman: Gates and Arne Love Big Data

By dianeravitch
May 16, 2014 //
24

Reader Laura Chapmam reminds us that the corporate-government combine wants Big Data. The demise of inBloom is only one stop in a long journey that invokes hundreds of millions of dollars and a foundational belief that what can be measure matters most:

Chapman writes:

The bare bones infrastructure for data-mongering was expanding in 1990, jump-started by a concerted effort to standardize vocabularies to characterize public education–think almanac–but expanded to fit the architecture of computer and information retrieval programs.
In tandem (as usual) Gates and USDE poured massive amounts of money into data-mongering starting in 2005, this intended to link student and teacher data in a continuum from birth to college and beyond.

Gates conjured the program called Teacher Student Data Link (TSDL), one facet of a data gathering campaign funded at $390,493,545 between 2005 and mid-May 2011 by the Gates’ Foundation.

This campaign envisions the link between teacher and student data serving eight purposes: 1. Determine which teachers help students become college-ready and successful, 2. Determine characteristics of effective educators, 3. Identify programs that prepare highly qualified and effective teachers, 4. Assess the value of non-traditional teacher preparation programs, 5. Evaluate professional development programs, 6. Determine variables that help or hinder student learning, 7. Plan effective assistance for teachers early in their career, and 8. Inform policy makers of best value practices, including compensation. See http://www.dataqualitycampaign.org/about

The TSDL system is intended to ensure that all courses are based on standards, and that all responsibilities for learning are assigned to one or more “teachers of record” in charge of a student or class. A teacher of record has a unique identifier (think barcode) for an entire career in teaching. A record is generated whenever a teacher of record has some specified proportion of responsibility for “a student’s learning activities” identified by the performance measures for a particular standard, subject, and grade level.

In addition to the eight purposes noted above, the TSDL system aims to have ”period-by-period tracking of teachers and students every day; including tests, quizzes, projects, homework, classroom participation, or other forms of day-to-day assessments and progress measures”—a level of accountability (I call it surveillance) that is said to be comparable to business practices (TSDL, 2011, “Key Components”).

This system will keep current and longitudinal data on teachers and individual students, schools, districts, states, and educators ranging from principals to higher education faculty. The aim is to determine the “best value” investments in education and monitor outcomes, taking into account as many demographic factors as possible, including health records for preschoolers. In Bloom may be dead but there are data-warehouses supported in part by Gates committed to that vision of data mining ( e.g. Battelle for Kids in Ohio).

On the federal side we have The Statewide Longitudinal Data Systems (SLDS) Grant Program, authorized under Title II, Educational Technical Assistance of the ‘‘Education Sciences Reform Act of 2002 H. R. 3801.” The first grants were made in 2005, the same year that the Gates’ Foundation started the parallel Data Quality Campaign.

See http://nces.ed.gov/programs/slds/
Achieve promoted, and still promotes, the Data Quality Campaign with a special focus on getting state policy makers to track individual students’ progress from pre-K to graduation and to use that data “to improve outcomes.” The program is being extended to teacher education with college programs measured by the test scores their graduates produce when they enter classrooms. See http://aacte.org/index.php?/Media-Center/AACTE-in-the-News/administration-pushes-teacher-prep-accountability.html.

In Bloom may be dead but all this other work is still in motion.

I think it wise to listen to some experts on Big Data. “We are more susceptible than we may think to the ‘dictatorship of data’—that is, letting the data govern us in ways that may do as much harm as good. The threat is that we will let our-selves be mindlessly bound by the output of our analyses even when we have reasonable grounds for suspecting something is amiss.

Or that we will attribute a degree of truth to data which it does not deserve.” Viktor Mayer-Schönberger & Kenneth Cukier. (2013). Big Data: A Revolution That Will Transform How We Live, Work, and Think. Boston: Houghton Mifflin. p. 166.

Categories Data and Data Mining, Duncan, Arne, Gates Foundation, Bill Gates

Anthony Cody: How Will Big Data Transform Education?

By dianeravitch
May 16, 2014 //
11

Anthony Cody recently read Simon Head’s Mindless: Why Smarter Machines Are Making Dumber Humans. This book, Cody says, shows how society is organized to benefit corporations, not people. He then includes a video clip from the CEO of Knewton, who claims that education is ripe for data mining.

He says:

“Education happens to be the world’s most data minable industry by far. And its not even close…. The name of the game is data per user. So one of the things that fakes us out about data in education is because it is so big – like the fourth biggest industry in the world – it produces incredible quantities of data. But data that just produces one or two data points per user per day is not really all that valuable to an individual user. It might be valuable to like a school district administrator, but maybe not even then. So let’s just compare. Netflix and Amazon get in the ones of data points per user per day. Google and Facebook get in the tens of data points per user per day. So you do ten minutes of messing around in Google and you produce about a dozen data points for Google. So Knewton today gets five to ten million actionable data points per student per day. Now we do that, because we get people, if you can believe it, to tag every single sentence of their content – we have a large publishing partnership with Pearson, and they’ve tagged all of their content. And we’re an open standard, so anyone can tag to us. If you tag all of your content, and you do it down to the atomic concept level, down to the sentence, down to the clause, you unlock an incredible amount of trapped, hidden data.

“We literally know everything about you and how you learn best. Everything. Because we have five orders of magnitude more data about you than Google has. We literally have more data about our students than any company has about anybody else, about anything, and it’s not even close. That’s how we do it.”

The day of Big Data grows closer. Arne Duncan sees it. David Coleman sees it. Do you see it?

Categories Common Core, Data and Data Mining, Education Industry, Education Reform

OECD Sponsors School Level Tests to Foster Test Mania and Its Own Control

By dianeravitch
May 11, 2014 //
20

The OECD has created tests that schools can administer to their students in order to compare them to the nations of the world.

Some schools have gleefully administered the tests, happy to discover how their students compare to children of the same age in the rest of the world.

Finnish educator Pasi Sahlberg, a visiting professor at Harvard Graduate School of Education this year, warned that it was not valid to compare schools to national systems.

The OECD test has sponsors now but it will eventually be a money-maker:

“Although these early administrations have been partly subsidized by private philanthropies, most districts will have to pay $11,500 per school in order to participate starting next year, according to Peter Kannam at America Achieves, a nonprofit that has been recruiting new schools and coordinating exchanges among participants.”

http://www.americaachieves.org/oecd#faq

“The development of this new diagnostic tool by the OECD was made possible by America Achieves, Bloomberg Philanthropies, Carnegie Corporation of New York, The William and Flora Hewlett Foundation, and the Kern Family Foundation. Additional support was provided by the Craig and Barbara Barrett Foundation, National Public Education Support Fund, the Stuart Foundation, and the Rodel Charitable Foundation of Arizona.”

Many educators can’t resist the temptation to administer yet another test. What would they do without data? Would they know how to diagnose children’s needs and plan for education without external tests to guide them? Surely, they cannot trust teachers to write their own tests or evaluate student needs.

In the ideal world of the future, school will be devoted entirely to testing, preferably to tests created solely by Pearson and/or the OECD. All learning will be standardized, and all children will be test-taking machines, programmed to find the right answer to every question. Th questions and the answers will be the sole property of Pearson and/or OECD.

Any learning not on the test will be considered a waste of time. Those who choose to think for themselves will be considered outliers, rebels, outcasts, possibly dangers to society. All “knowledge” will be strictly monitored by the Pearson/OECD bureaucracy.

The rules of life in this new society will be:

“We measure what we treasure.”
“You can’t control what you can’t measure.”
“Whatever cannot be quantified does not matter.”
“All problems can be solved by measurement and data.”
“Test scores determine one’s life potential.”
“Test scores are the best measure of students, teachers, and schools.”

Welcome to our Brave New World.

Categories Data and Data Mining, Education Industry, International, Standardized Testing, Testing

Oklahoma: Ms. Bullen’s Data-Rich Year

By dianeravitch
May 6, 2014 //
36

Blue Cereal Education is the name of an educator-blogger in the Tulsa area. He or she has helpfully reproduced a graphic from the website of the Oklahoma State Department of Education that will show you, in a flash, how teaching and learning are being systematically destroyed in this country by robots who pretend to be humans.

It is called “Ms. Bullen’s Data-Rich Year,” but it might as well be called “The End of Teaching as We Know It As we Collect Data and Pretend It Matters.”

Here are a few of the 15 steps to a data-rich classroom:

“(7) You are expected to create an IEP for each and every one of your students before school even begins! (Step Two) Setting aside the fact that this is insane, it’s still nine full steps before Step Eleven, where an ‘early warning system’ (which appears to be an iPad app) will send an alert to a strange man in the room that Joey is off-track, or failing. Presumably the strange man will tell Ms. Bullen, who can call Joey’s very involved parents in to look at the full-sized mural she’s devoted to the Chutes & Ladders version of Joey’s educational journey. Thank god there’s finally a way to know when students are failing – other than the fact that they’re, for example, failing.

“(8) You are expected to immediately discard the approximately 170 IEP’s you’ve spent weeks creating so you can “adjust instruction on the fly” (Step Three) based solely and exclusively on the perceived reactions of Joey. We can only hope the 34 other students in the room are not offended at the impact this must have on their individualized learning experience. At the same time, this is a great moment – it’s the only point in All 18 Steps that assumes for even an instant that you (represented here by Ms. Bullen) have any idea what you’re doing without consulting a few dozen spreadsheets of data. But don’t worry – you won’t be stuck teaching ‘on the fly’ for long!

“(9) You will have plenty of time to meet one on one with each of your students (Step Six) to discuss their behavior, attendance data (which is different from attendance… how?), and performance, as well as what Joey’s parents want for him – during the one moment in which is overly involved parents are conspicuously absent. You’ll set some individualized goals for the year to replace that IEP you developed before you met him, then threw out in Step Three.

“Assuming you have approximately 168 students, and that each of these meetings take about 10 minutes, that’s only about… 28 hours each week. Or is it each month? I’m not sure how often this one is supposed to happen. Let’s assume it’s just once – it’s not like Joey’s performance, behavior, goals, or attendance are likely to change throughout the year. So we’ll just use that extra 28 hours floating around during, say… October. Nothing that important happens in October anyway.

“(10) I’m not sure what “Data Coaches” are (Step Seven), although each school apparently has several (they must share office space with all the Tutors and Trainers – no wonder Oklahoma schools are so darned inefficient with how they spend district money!) Apparently while teachers celebrate their one collective decent idea, the Data Coaches do some sort of ceremonial handshake – or perhaps it’s a dance. I’m not familiar with that culture, but I’d really like to see that. There simply aren’t enough dances based on hard educational data.”

Now that is only four of the 15 steps that the State Education Department includes in its graphic.

Taken together, the graphic demonstrates a system that cares nothing about education, nothing about children, and nothing about teachers.

Perhaps it was put together by a computer or by someone who wants to promote home schooling.

Categories Data and Data Mining, Oklahoma
« Older Entries
Newer Entries »