Archives for category: Data

When I was a young historian, back in the 1970s, I would occasionally search for a fact about American education in the nineteenth or early twentieth century to help me write an article or book. There was no Internet. I wasn’t sure which books had the right statistics. So I invariably called the National Center for Education Statistics (NCES), which is the statistical arm of the U.S. Department of Education (actually there was no Department of Education until 1980 [Congress passed the legislation in 1979, and the Department became operational in 1980]; the NCES was the longstanding research and statistics arm of the U.S. Office of Education). The federal role in education began in 1867 under President Andrew Johnson with the creation of a Department of Education, whose sole mission was to collect and disseminate information on the condition and progress of education in the United States. In 1868, however, due to fears that the new Department might eventually seek to exert control over state and local education policy, the Department was demoted to the U.S. Office of Education. Its central purpose, the collection and dissemination of accurate information, is today the role of the NCES.


When I called for information, there was one person who knew where to find whatever I was looking for. Not opinion or interpretation, just the facts. His name was Vance Grant. He invariably took my calls and just as invariably found the answer, if it existed in federal records.


In 1991, I became Assistant Secretary in charge of OERI (the Office of Education Research and Improvement) and NCES was part of my agency–the most important part. I met Vance Grant, and I had an idea. Why not assemble all the historical data into a publication? With the help of the very able career staff at NCES, especially Tom Snyder and Vince Grant, and with the help of historian Maris Vinovskis, who had taken a leave at my request from the University of Michigan to work with OERI staff, the publication became a reality.


It is called 120 Years of American Education: A Statistical Portrait.


I can say now in retrospect that this publication was the most useful thing I did during my two years in the federal government.


You too can browse its pages and charts and graphs via the Internet to see the growth of education in the United States.


Although not many people know of its existence, it is still the only reliable source of historical data on American education.

The Hechinger Report describes a study from an online company that has been data mining the kids.

Based on data mining and test scores, it reaches the conclusion that only 4.7 minutes of additional reading is enough to lift kids at the bottom to the top half!

Please, what is .7 of a minute?

Dad Gone Wild is a blogger in Nashville whose children attend the local public schools. In this post, he recounts his decision to start serious running, first for his health, but then because he became obsessed with collecting data about his running.

You see, it started with a simple app that measured time and distance and kept a running total for a benchmark. But then it progressed to enough of a dependency to justify getting a top-of-the-line Garmin race watch because, well, dependence on data requires more data. Where I once was only concerned with how far and how fast I ran, I am now measuring footfalls, cadence, and several other categories that a) I don’t know exactly what they mean, and b) I wouldn’t know how to change them even if I knew what they meant. I imagine that my running has gotten better over the years, but I attribute that to running more and trying to eat better, not measuring my cadence and footfalls. My sense of accomplishment has certainly not grown; in fact, I’ve noticed a weird phenomenon.

If I head out on a run and one of my measurement tools isn’t working, I’ll either quit the run or proceed without the measurement. The weird part is, that if I continue the run sans measurement, it’s almost like it didn’t happen. I’m going the same distance. I’m burning the same miles, but for some reason it doesn’t feel real. When I look at the data for the month, those runs cease to exist and any progress I might have made is discounted. When I compare my results to friends I “compete” against, it almost feels like I’m lying about those results.

He realizes that the data have taken over; he no longer thinks about why he runs, but only about the measurements. That’s data addiction.

We also say that standardized tests are for the good of the child and that we are preventing minority children from slipping through the cracks, the opposite is actually true, but I ask you, how many people have cited that one test back in 7th grade that opened the door to learning for them? Until they took that test they were lost in the wilderness, but that test inspired them to greatness. Now substitute test with book or mathematical concept, and I bet you get a different answer. By putting emphasis on learning for the measurable we are actually restricting people’s potential. Like when I fail to take a 3-mile run because it doesn’t add significantly to my monthly totals, similarly, students will potentially fail to partake in opportunities to learn because it will produce no measurable results.

Data addiction also leads to putting undue pressure on suppliers. Let’s face it, that is what we are turning our teachers into, data suppliers. Our teachers are under a constant barrage to deliver more data. They are losing valuable time and sanity trying to meet the ever-increasing need. They are in an endless churn to produce more data under the threat that if they don’t, we will replace them with people who can, though we never mention a viable source for these replacements. Perhaps there is a teacher orchard producing an overabundance of quality teachers willing to work for decreasing pay and autonomy that I am unaware of.

Our thirst for more that is measurable has reduced the art of teaching into that of a producer. We’ve lost sight that teaching children is more than just about instruction in reading, writing, and arithmetic. Think about the teachers who had a profound effect on you and your growth. Do you remember them because in fifth grade they had you reading on a seventh grade level, or is it deeper than that? Is it because you felt that they cared for you and were truly vested in helping you understand your place in the world?

Can data addiction be cured or are we doomed to reduce everything we do to numbers, forgetting why we do those things?

We will have a life with data, but a life without purpose.

A reader of a post this morning about a letter from John Kline to Arne Duncan asked for more information about the Department of Education’s change of regulations governing FERPA (the Family and Educational Rights and Privacy Act of 1974).

In a post two years ago, I described the lawsuit filed by EPIC (the Electronic Privacy Information Center), which sought to block the changes in federal regulations in 2011 that loosened the protections of student privacy.

Here is an explanation of the lawsuit that appeared on Valerie Strauss’s Answer Sheet blog.

The EPIC lawsuit was dismissed in 2013; the Court held that EPIC did not have standing to sue. Its ruling did not deal with the substantive claims.

Parent groups became concerned about FERPA when the Gates Foundation and the Carnegie Corporation funded the “Shared Learning Collaborative,” which was renamed inBloom. The plan was to aggregate personally identifiable student data from state data warehouses, store them in a cloud, and make them available for use by others. Whether those others included vendors, researchers, or commercial enterprises is not sure, but parents vehemently opposed the entire plan. The software was developed by Rupert Murdoch’s Wireless Generation (part of Joel Klein’s Amplify division) and the data would be stored in a “cloud” managed by amazon. Parent groups, fearful that their child’s personal data would be mined, testified against the data-sharing agreements in every state and district that agreed to join inBloom, and the effort collapsed. The last state to withdraw was New York, because Commissioner John King supported inBloom. The legislature compelled the state’s withdrawal. When there were no states or districts willing to share student data, inBloom had no reason to exist.

The organization to fight inBloom was led by Leonie Haimson of New York and Rachel Strickland of Colorado, who formed the Parent Coalition for Student Privacy. See here and here and here.

One of the goals of Race to the Top was to create a national student data base, one collected from every state. Creating such a data base was one of the conditions of eligibility for Race to the Top. One of the companies formed to mine the data was created by the Gates Foundation and the Carnegie Corporation and called inBloom. InBloom intended to use software created by Wireless Generation (part of Joel Klein’s Amplify, owned by Rupert Murdoch) to aggregate this personally identifiable information and put it into the “cloud,” that place in cyberspace where all data lives forever. There would have been 400 data points for each student. Parents organized in states and districts to prevent this breach of their children’s privacy, and state after state, district after district, dropped out because the plan was indefensible (New York state, led by John King, was the last to drop out, and then only because the Legislature commanded him to do so).

Now a letter has become public, from John Kline to Arne Duncan, warning Duncan about the dangers of creating a national data base of student information.

There is a federal law called FERPA that is supposed to protect the privacy of individual students, but Duncan changed the regulations to make data mining possible.

Peter Greene notes that the corporate reformers are still pressing for more data on each student. There can never be enough data. When there is more than enough, then you have Big Data, where government and corporations can analyze mega-trends. But reformers don’t say that this is what they want; they insist that this data is what parents want and need, even if they don’t say so themselves.

He writes:

Over at Getting Smart, a website devoted to selling educational product, guest writer Aimee Rogstad Guidera makes her case for more data collection for each student– because it’s what parents want.

“Parents are eager for information about their child’s education. As a mom, I want to know if my daughter is struggling in math before she comes home in tears. I need information to support my child’s learning at home, and to support my child and her teacher in making the best decisions for her learning in the classroom.”

Maybe I just don’t get it, but I’m inclined to think that if you didn’t know your child was having trouble in math before the coming-home-in-tears part, you’re just not paying attention. I have heard this pitch enough times to make me occasionally wonder if there is, in fact, some place where teachers keep every scrap of information carefully hoarded, students never speak to their parents about school, parents never ask about school, and all parent requests for conferences and information are denied by all school personnel. Maybe there is some place where parents are so deeply clueless and helpless that they have no idea how their students are doing.
Or maybe Guidera is the CEO and President of the Data Quality Campaign, a group interested in student data and funded by the Gates Foundation, the Waltons, the Dells, and the Ford Foundation. They do have some rules about how such data should be kept in a safe lockbox, but they are clearly Big Data fans.

Guidera is advocating for student data backpacks– little (or not so little) bundles of data that just follow students around, providing parents with all sorts of longitudinal data (because, again, parents don’t know much about their own children).

Greene has some advice for parents who want more information about how their child is doing: pick up the phone and call the teacher.

Julian Vasquez Heilig collected data on New Orleans and Louisiana and wondered what the hullaballoo was about. The state is one of the lowest-performing in the nation, by federal measures; and the charter schools have produced mediocre results.

Heilig’s policy brief was sponsored by the Network for Public Education. Since NPE supports public schools, it is hardly surprising that it looks with disfavor on a massive experiment in privatization. Every high-performing nation in the world has an equitable public school system. We should too.

The report examines NAEP scores, ACT scores, high school graduation rates, dropout rates, AP course taking rates, and other criteria.

A useful conclusion to a day of all-New Orleans, all-the-time.

You might want to refer to this policy brief when your legislator or Governor offers a proposal for an “achievement school district” or an “opportunity school district” modeled on New Orleans “Recovery School District.”

Gene V. Glass is one of our mation’s superstar researchers of education. His field for many decades was measurement. He describes how hopeful the field was that better measurement of students would solve important problems.

But in this post, he explains that he is resigning from his field. Measurement has over promised and under delivered.

“The degrading of public education has involved impugning its effectiveness, cutting its budget, and busting its unions. Educational measurement has been the perfect tool for accomplishing all three: cheap and scientific looking….

“Teachers and many parents understand that children’s development is far too complex to capture with an hour or two taking a standardized test. So resistance has been met with legislated mandates. The test company lobbyists convince politicians that grading teachers and schools is as easy as grading cuts of meat. A huge publishing company from the UK has spent $8 million in the past decade lobbying Congress. Politicians believe that testing must be the cornerstone of any education policy.

“The results of this cronyism between corporations and politicians have been chaotic. Parents see the stress placed on their children and report them sick on test day. Educators, under pressure they see as illegitimate, break the rules imposed on them by governments. Many teachers put their best judgment and best lessons aside and drill children on how to score high on multiple-choice tests. And too many of the best teachers exit the profession.

“When measurement became the instrument of accountability, testing companies prospered and schools suffered. I have watched this happen for several years now. I have slowly withdrawn my intellectual commitment to the field of measurement. Recently I asked my dean to switch my affiliation from the measurement program to the policy program. I am no longer comfortable being associated with the discipline of educational measurement.”

This article on “The Costs of Accountability” appeared in The American Interest. It was written by Jerry Z. Muller, a professor of history at Catholic University of America in Washington, D.C. It is a long and thoughtful article, and I can offer just a few snippets. I urge you to read it. It is a five-star article that explains how much money and energy is wasted in pursuit of the Golden Fleece of “accountability.” It has become an industry unto itself.

He begins:

The Google Ngram Viewer, which instantly searches through thousands of scanned books and other publications, provides a rough but telling portrait of changes in our culture. Set the parameters by years, type in a term or phrase, and up pops a graph showing the incidence of the words selected from 1800 to the present. Look up “gender”, for example, and you will see a line that curves upward around 1972; the slope becomes steeper around 1980, reaches its peak in 2000, and afterwards declines gently. Type in “accountability” and behold a line that begins to curve upward around 1965, with an increasingly steep upward slope after 1985. So too with “metrics”, whose steep increase starts around 1985. “Benchmarks” follows the same pattern, as does “performance indicators.” But unlike “gender”, the lines for “accountability”, “metrics”, “benchmarks”, and “performance indicators” are all still on the upswing.

Today, “accountability” and its kissing cousins “metrics” and “performance indicators” seem to be, if not on every lip, then on every piece of legislation, and certainly on every policy memo in the Western world. In business, government, non-profit organizations, and education, “accountability” has become a ubiquitous meme—a pattern that repeats itself endlessly, albeit with thousands of localized variations.

The characteristic feature of the culture of accountability is the aspiration to replace judgment with standardized measurement. Judgment is understood as personal, subjective, and self-interested; metrics are supposed to provide information that is hard and objective. The strategy behind the numbers is to improve institutional efficiency by offering rewards to those whose metrics are highest or whose benchmarks have been reached, and by punishing those who fall behind relative to them. Policies based on these assumptions have been on the march for decades, hugely enabled in recent years by dramatic technological advances, and as the ever-rising slope of the Ngram graphs indicate, their assumed truth goes marching on.

The attractions of accountability metrics are apparent. Yet like every culture, the culture of accountability has carved out its own unquestioned sacred space and, as with all arguments from presumed authority, possesses its characteristic blind spots. In this case, the virtues of accountability metrics have been oversold and their costs are underappreciated. It is high time to call accountability and metrics to account.

That might seem a quixotic, if not also a perverse, aspiration. What, after all, could be objectionable about accountability? Should not individuals, departments, divisions, be held to account? And how to do that without counting what they are doing in some standardized, numerical form? How can they be held to firm standards and expectations without providing specific achievement goals, that is, “benchmarks”? And how are people and institutions to be motivated unless rewards are tied to measureable performance? To those in thrall to the culture of accountability, to call its virtues into question is tantamount to championing secrecy, irresponsibility, and, worst of all, imprecision. It is to mark oneself as an enemy of democratic transparency.

To be sure, decision-making based on standardized measurement is often superior to judgment based on personal experience and expertise. Decisions based on big data are useful when the experience of any single practitioner is likely to be too limited to develop an intuitive feel for or reliable measure of efficacy. When a physician confronts the symptoms of a rare disorder, for example, she is better advised to rely on standardized criteria based on the aggregation of many cases. Data-based checklists—standardized procedures for how to proceed under routine or sometimes emergency conditions—have proven valuable in fields as varied as airline operation, rescue squad work, urban policing, and nuclear power plant safety, among a great many.

Clearly, the attempt to measure performance, however difficult it can be, is intrinsically desirable if what is actually measured is a reasonable proxy for what is intended to be measured. But that is not always the case, and between the two is where the blind spots form.

Measurement schemes are deceptively attractive because they often “prove” themselves through low-hanging fruit. They may indeed identify and help to remedy specific problems: It’s good to know which hospitals have the highest rates of infections, which airlines have the best on-time arrival records, and so on, because it can energize and improve performance. But, in many cases, the extension of standardized measurement may suffer diminished utility and even become counterproductive if sensible pragmatism gives way to metric madness. Measurement can readily become counterproductive when it tries to measure the unmeasurable and quantify the unquantifiable, whether to determine rewards or for other purposes. This tends to be the case as the scale of what is being measured grows while the activity itself becomes functionally differentiated, and when those tasked with doing the measuring are detached organizationally from the activity being measured.

He writes specifically about education:

No Child, Doctor, or Cop Left Behind

In the public sector, the show horse of accountability became “No Child Left Behind” (NCLB), an educational act signed into law with bipartisan support by George W. Bush in 2001 whose formal title was, “An act to close the achievement gap with accountability, flexibility, and choice, so that no child is left behind.”

The NCLB legislation grew out of more than a decade of heavy lobbying by business groups concerned about the quality of the workforce, civil rights groups worried about differential group achievement, and educational reformers who demanded national standards, tests, and assessment. The benefit of such measures was oversold, in terms little short of utopian.

Thus William Kolberg of the National Alliance of Business asserted that, “the establishment of a system of national standards, coupled with assessment, would ensure that every student leaves compulsory school with a demonstrated ability to read, write, compute and perform at world-class levels in general school subjects.” The first fruit of this effort, on the Federal level, was the “Improving America’s Schools Act” adopted under President Clinton in 1994. Meanwhile, in Texas, Governor George W. Bush became a champion of mandated testing and educational accountability, a stance that presaged his support for NCLB.

Under NCLB states were to test every student in grades 3–8 each year in math, reading, and science. The act was meant to bring all students to “academic proficiency” by 2014, and to ensure that each group of students (including blacks and Hispanics) within each school made “adequate yearly progress” toward proficiency each year. It imposed an escalating series of penalties and sanctions for schools in which the designated groups of students did not make adequate progress. Despite opposition from conservative Republicans antipathetic to the spread of Federal power over education, and of some liberal Democrats, the act was co-sponsored by Senator Edward Kennedy and passed both houses of Congress with majority Republican and Democratic support. Advocates of the reforms maintained that the act would create incentives for improved outcomes by aligning the behavior of teachers, students, and schools with “the performance goals of the system.”

Yet more than a decade after its implementation, the benefits of the accountability provisions of NCLB remain elusive. Its advocates grasp at any evidence of improvement on any test at any grade in any demographic group for proof of NCLB’s efficacy. But test scores for primary school students have gone up only slightly, and no more quickly than before the legislation was enacted. Its impact on the test scores of high school students has been more limited still.

The unintended consequences of NCLB’s testing-and-accountability regime are more tangible, however, and exemplify many of the characteristic pitfalls of the culture of accountability. Under NCLB, scores on standardized tests are the numerical metric by which success and failure are judged. And the stakes are high for teachers and principals, whose salaries and very jobs depend on this performance indicator. It is no wonder, then, that teachers (encouraged by their principals) divert class time toward the subjects tested—mathematics and English—and away from history, social studies, art, and music. Instruction in math and English is narrowly focused on the skills required by the test rather than broader cognitive processes: Students learn test-taking strategies rather than substantive knowledge. Much class time is devoted to practicing for tests, hardly a source of stimulation for pupils.

Even worse than the perverse incentives involved in “teaching to the test” is the technique of improving average achievement levels by reclassifying weaker students as disabled, thus removing them from the assessment pool. Then there is out-and-out cheating, as teachers alter student answers or toss out tests by students likely to be low scorers, phenomena well documented in Atlanta, Chicago, Cleveland, Houston, Dallas, and other cities. Mayors and governors have diminished the difficulty of tests, or lowered the grades required to pass the test, in order to raise the pass rate and thus demonstrate the success of their educational reforms—and get more Federal money by so doing.

Another effect of NCLB is the demoralization of teachers. Many teachers perceive the regimen created by the culture of accountability as robbing them of their autonomy, and of the ability to use their discretion and creativity in designing and implementing the curriculum. The result has been a wave of early retirements by experienced teachers, and the movement of the more creative ones away from public and toward private schools, which are not bound by NCLB.

Despite the pitfalls of NCLB, the Obama Administration doubled down on accountability and metrics in K-12 education. In 2009, it introduced “Race to the Top”, which used funds from the American Recovery and Reinvestment Act to induce states “to adopt college- and career-ready standards and assessments; build data systems that measure student growth and success; and link student achievement to teachers and administrators.” This shows what happens these days when accountability metrics do not yield the result desired: Measure more, but differently, until you get the result you want.

Metric madness is not limited to education. Some of the problems evident in NCLB pop up in fields from medicine to policing.

We have often heard that Mark Twain said that there are “lies, damned lies, and statistics.” I checked with Wikipedia, and it turns out that this phrase has many fathers. For example, says Wikipedia:

Mark Twain popularized the saying in Chapters from My Autobiography, published in the North American Review in 1906. “Figures often beguile me,” he wrote, “particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: ‘There are three kinds of lies: lies, damned lies, and statistics.'”

But there are other claimants to the phrase, as the article notes, including one who ranked false statements as “a fib, a lie, and statistics.” A variation on this phrase is: “simple liars, damned liars, and experts.”

And then we come to the “New Orleans Miracle.” According to recent research, test scores have improved dramatically since 2005, when Hurricane Katrina wiped out the public school district and replaced it with a district which is almost all-charter. Douglas Harris, director of the Research Alliance in New Orleans reported the results in the conservative journal Education Next, which promotes alternatives to public education. Bottom line, in his account: Wiping out the district, firing all the teachers, wiping out the union contract, hiring Teach for America to replace veteran teachers, has mostly good outcomes. Education Week reported Harris’s claim of dramatic progress.

But then there is Mercedes Schneider, who reports that the state released 2015 ACT scores for every district, and the New Orleans Recovery School District ranked 70th out of 73 districts in the state. Its ACT scores are virtually unchanged over the past three years. The RSD ACT scores of 16.6 are far below the state average of 19.4.

An average ACT score of 16.6 is low. Louisiana State University requires a composite score of 22. A composite of 20 qualifies for La’s tuition waiver to a 4-year institution; a composite of 17 qualifies for tuition waiver for 2-year technical college.

And here’s the latest study by Research on Reforms in New Orleans, comparing the Orleans Parish public schools to the reformers’ Recovery School District. “A study of three ninth grade cohorts, beginning with the 2006-07 year, shows that the percentage of OPSB 9th graders who graduate within four years is almost double that of RSD 9th graders, and the RSD’s dropout rate is nearly triple that of the OPSB.”

You may decide which statistics matter most to you. But whichever you choose, be sure to read Jennifer Berkshire’s account of what the reforms in New Orleans have produced. It is important context in which to place whatever data you think is most valuable.

Paul Thomas reviews the debate about The progress of Néw Orleans and concludes:

“So we are left with two truisms about education publications and education reform: (1) If “Education” is in the publication title, you better do your homework, and (2) if education reform is touted to achieve outcomes that seem too good to be true, then they likely aren’t true.”


Get every new post delivered to your Inbox.

Join 162,959 other followers