Jay Mathews invented a high school ranking program that relies on certain criteria, especially the number of students who take AP and IB tests. The data are self-reported by the schools.
Carol Burris warned Jay that some of his rankings were highly improbable, especially those reported by the IDEA charter chain. She was right. Carol thinks that the temptation to “juke the stats” is too powerful. Jay thinks that there was an honest error.
Jay writes:
“When I started what The Washington Post now calls the America’s Most Challenging High Schools list, I was told not to trust data from schools and school districts. They’re sloppy and sometimes dishonest, people said. It won’t work.
“That was 19 years ago. The doubters were wrong. The educators I deal with have proved to be unfailingly honest. Mistakes are rare. But the biggest so far just happened. The IDEA Public Schools charter network in Texas told me it provided incorrect numbers of Advanced Placement tests at six of its schools for the 2017 list published in May.
“As a result, the five IDEA schools that were in the top 10 have dropped several places on the corrected list. “We messed up,” said IDEA founder and chief executive Tom Torkelson.”
Actually, IDEA has a documented history of overstating its success. Ed Fuller, a professor at Penn State formerly at the University of Texas, debunked the IDEA claim that 100% of its graduates were accepted into four-year colleges.
Ed Fuller wrote on his blog:
“IDEA Charter School markets itself as a college preparatory education organization with goals of enrolling 100% of graduates in four-year universities and have 100% persistence and graduation rates in college.[1] Indeed, in the introduction of the most recent annual report, Dr. Tom Torkelson, CEO of IDEA Public Schools, makes the following statement:
“IDEA puts students on path to succeed in an increasingly competitive global marketplace by providing a rigorous college preparatory education and preparing our low-income, Hispanic and minority students in under-served communities across Texas to apply, matriculate and succeed in a the four year college or university of their choice. To date, 100% of IDEA graduates have been accepted to a four-year college or university and our student (sic) are demonstrating remarkable staying power: 92% are either still in college or have graduated.[2]
“Further, on page six of the report, the claim is made that, “ . . . for the fifth year in a row, 100% of IDEA graduates enrolled in four-year colleges and universities, fulfilling IDEA’s mission of College For All Children.”[3]
“Yet, these claims are demonstrably false, the report fails to cite any data sources or studies that substantiate the claims, and the report fails to report publicly available data on the performance of students in four-year universities. The remainder of this short report substantiates my claims made about statements included in the IDEA annual report and provides data from the Texas Higher Education Coordinating Board about the Performance of IDEA Public School graduates in Texas four-year universities.”
Fuller goes on to show that 100% of IDEA graduates did not enroll in four-year colleges, and many of these students performed very poorly in college.
Given IDEA’s history of boasts and overstatements, why should Jay Mathews accept its self-reported data?
How many other self-reported errors are hidden in those rankings? Carol wonders if Jay is rewarding charters that push kids out who can’t or won’t take the AP courses. She thinks he should create two separate lists: one for charters, which have the power to choose and exclude their students; another for public schools that accept all students. I would suggest a third category: selective public schools. How can anyone fairly compare a comprehensive high school that accepts all who apply, with a selective high school that admits only those who pass an examination? How can anyone fairly compare charter schools with high attrition rates–weeding out low performers– to those public schools that enroll students with disabilities and students who are learning English, as well as children who are homeless.
I don’t see the purpose of ranking selective high schools and non-selective high schools. I’m having trouble understanding the value of the rankings no matter how they are reported.
Diane Ravitch's blog 
This is terrifying. Isn’t separation of church and state in the Constitution? It seems to me that this makes vouchers for any kind of school Constitutional. If that is the case, either taxes will have to sky rocket or nobody will have any kind of quality education. And Kennedy might retire?
Sorry meant this for the Supreme Court post.
You really have to delve into the details to get the whole story, and since 99% of people won’t bother, it doesn’t matter.
Ed reformers regularly compare one chain of charter schools in Cleveland to Cleveland Public Schools. But charters draw from outside Cleveland. It’s a bogus comparison, but they ALL do it, from Fordham to Duncan to Obama.
It mostly bothers me from Fordham, because they know better. I don’t think Duncan had a clue how Cleveland charter schools work.
Free markets are efficient and if markets are left alone, the consumer will always make the right choice based on the evidence of who has the best products at the right price.
Diane I have this bridge near your house I am looking to sell do you know any buyers
.
Is it a highly ranked bridge?
If the criteria for ranking ( number of students taking AP classes and tests) are bogus, its completely irrelevant whether the reported data (number of AP tests) are correct.
It’s garbage in/ garbage out.
Sí, señor.
It’s a completely bogus list. If it isn’t, then the same standard can be applied to high stakes tests: measure teacher effectiveness by the % of students taking the test.
The number of current year tests is just one of the statistics that can be gamed. So, too, the % of students eligible for free or reduced cost lunch.
Precisely why his rankings or any rankings are of any substance or of any good: “Jay Mathews invented a high school ranking program that relies on certain criteria, especially the number of students who take AP and IB tests.” Those criteria are not what makes schools great places for kids.
I wonder why he made those so important, especially AP tests….LOL…. We know.
School rankings are stupid.
Sí, señora.
The same can be said for typical public schools as well.
Especially regarding the numbers of students that apply and how ready they may not actually be when they get there.
Many need remedial classes even after being on honor roll in high school and many don’t make it through the first year, let alone they college to graduate.
The key phrase: “how ready they may not actually be when they get there.”
This readiness starts in kindergarten and most of the children that start behind stay behind and fall further behind as they progress k-12. Most of these children also come from home environments where no one reads even if they can read. There are no books, no magazines, no newspapers and the child never sees his/her adult role models reading anything. They start out in a literate wasteland and, outside the school, grow up in one.
And this isn’t just in the U.S. It is everywhere but up until NCLB, the U.S. was closing that gap while countries that often rank higher on the International PISA tests saw that gap widening.
Stanford Report, January 15, 2013.
“There is an achievement gap between more and less disadvantaged students in EVERY country; surprisingly, that gap is smaller in the United States than in similar post-industrial countries, and not much larger than in the very highest scoring countries.
“Achievement of U.S. disadvantaged students has been rising rapidly over time, while achievement of disadvantaged students in countries to which the United States is frequently unfavorably compared – Canada, Finland and Korea, for example – has been falling rapidly.”
http://news.stanford.edu/news/2013/january/test-scores-ranking-011513.html
I’m sure that Jay Matthews also thinks Donald Trump is the most successful, smartest, and most honest man (etc., etc., etc. forever) in the world because Trump said so.
And yet all public schools are still required to provide all students with more than a deminimis education.
I have no idea what your point is. Explain, please.
It was in response to LL’s statement:
“This readiness starts in kindergarten and most of the children that start behind stay behind and fall further behind as they progress k-12. Most of these children also come from home environments where no one reads even if they can read. There are no books, no magazines, no newspapers and the child never sees his/her adult role models reading anything. They start out in a literate wasteland and, outside the school, grow up in one.”
So using this logic, these students must therefore be beyond help and apparently are destined to continue the generational cycle of illiteracy & innumeracy struggles, and such lack of preparation is a deal breaker and hence schools should be exempt from accountability to improve and teach these at risk students.
But under the law of the land, including the most recent SCOTUS ruling, provide them an education that is more than deminimus and public schools are still required to provide FAPE for ALL students, no matter their skill level upon entering school.
It means that students should be receiving an academic benefit to an extent that matches their cognitive abilities. And it should include closing the gaps they have and not letting them fall further and further behind each passing year, as well as whatever is needed to level the playing field, in parallel with providing effective remedial instruction.
Plus, all public schools know that this population of students exist, yet they do barely anything to close the gaps they have and to break the cycles these children and families present at school with regarding helping them to effectively overcome their personal struggles with reading, writing and math.
Given the fact that most of these children are attending underfunded, poorly resourced public schools that are under draconian strictures as to how and what they can do in a classroom, I find it rather simplistic to claim that public schools are doing nothing. It is frankly amazing to me when I hear of the successes in spite of the seemingly purposeful efforts to undercut them. I know there is room for improvement even under the bare bones budgets under which most of these schools operate. However, the responsibility for providing an education that meets the needs of the students requires acknowledgement from the establishment that scapegoating the schools for the failures of society in general is neither fair or productive. We have a model of community schools that seems to be successful, but we need the support from the whole community and all funding sources. I was a teacher. I could not meet all the needs of my students nor I should I have been expected to. I can’t correct the health issues, housing problems, family struggles, and/or food deficits to name a few areas that impacted my students. I won’t even touch the difference in resources between that school and the schools my children attended in a far more affluent community. It’s way past time to drop the simplistic scapegoating and define the problems facing people living in poverty from a societal perspective. It is not and never was primarily and education issue. Let teachers do their job, which has nothing to do with training kids to “crush” high stakes tests. I have never met a child who was well defined by his/her test scores, and yet that is what the corporate model demands. At what point will we question a model that “efficiently ” defines educational goals through numbers and yet consistently produces poor results?
@Speculate:
And how did you treat parents who had the data showing their children needed programs you offered in your district but that were refused to them? For instance Wilson Reading which your school advertised on it’s webpage?
And what if your school was one that had a very low student to teacher ratio?
And what if it was not an issue of funding because the child was a foster student and your school was being reimbursed and not one dime of the district’s funds were used and this child was actually more of a cash cow to your district by bringing in additional Title 1 funds because the student was in foster care?
You have lost me again. I have no idea what you are talking about. Who is @Speculate?
Speduktr,
I think that is you, with spelling changed by autocorrect
This actually demonstrates what is so wrong about the current “official” approach to education policy in the US.
Allowing a “columnist” to unilaterally decide the criteria for what makes a “challenging” high school and then come up with his own “ranking system” is absurd.
And actually, it’s not just eduction policy that columnists have unjustified influence over.
It’s foreign policy, economics, and a whole slew of other things.
It’s bizarre that some who has never really DONE anything other than write columns would be listened to by anyone, to say nothing of by people in positions of power.
SDP,
RIGHT. Why should schools, universities, professions, etc be judged by US News & World Report?
They should be rating restaurants.
And when they rate restaurants, at least someone goes and samples the food.
What do our community-based, democratic, transparent, non-profit, traditional public schools get – schools that once were the envy of the world and studied by many countries around the world?
Flawed, secretive tests that profit huge corporations like the UK’s Pearson – tests that are then used to judge those schools, teachers, and students and the test makers never visit any of those schools.
And if they did visit one or more of those schools, it would be a quick walk through – blind, deaf and dumb – ending in a press conference where there would be nothing but criticisms. These pirates, these liars, these frauds that worship at the altar of avarice would never spend enough time to learn what actually goes on in those classrooms or find out what the lives of those children taking the tests are like.
Mr. Mathews seems an open sort of guy, so if he’s reading comments, here’s mine:
After watching my kid do the AP track in high school my opinion is that more work for more work’s sake is not a good definition of challenge. The real challenge would be to say no to the AP classes and go do something meaningful.
Perhaps there was a time when AP credits were universally valuable in college. That no longer seems to be the case. Most selective colleges we researched do not take AP credits outright and most put limits on their uses. Some only take a limited number. Others allow students to use APs to skip a level, but students still have to take the same number of originally required credits to graduate. So where is the benefit to the student after all the crammed, pressured work? Why not simply take college courses in college where they belong? What’s the hurry?
High school AP tests cost about a hundred dollars a piece, plus the cost of review materials or books. This means that test completion after the course and test success is heavily weighted in favor of those with sufficient funds. After a year of paying hundreds for tests, this year our family said forget it. Take the class if you have to, skip the test. APs push privatization into public school.
Remember, kids take APs at year end, along with SATs, ACTs, exit exams, PARCC tests, final projects, sports season finals, family and some job obligations, etc. The extra work, stress, diversion of time from other pursuits to AP testing and high cost, with little or no lessening of college expense or work load, make AP’s a bad investment all around. WWhat’s the hurry? Who is benefitting from these classes?
Seems to me the College Board is raking it in off the backs of our kids who could be doing something better with their time. And sorry to be so direct, but doesn’t Mr.Mathews have a pretty big stake in perpetuating the AP system?
I would like to see Mr.Mathews do a full review of colleges’ views of APs, the cost of APs to students, and the (corrosive) impact APS have on high school learning–economic, segregative, time lost to other pursuits, class time and high school resources lost to other high school curriculum, undue stress. I’d like to see an analysis of who benefits financially and how much the College Board makes. How much money do corollary businesses like prep book manufacturers and folks like Mr. Mathews make from the AP tests?
I’d also like to see him think about other measures that indicate “challenge,” by which I’m assuming he means opportunities for deep intellectual stimulation. How about which high schools have the lowest student-teacher ratio, student driven curriculum, curriculum integrating subject matters, hands on classes, classes allowing deep dives into subjects according to student interest, classes requiring students to engage in self examination, to support one another?
I think APs are a money generating rat race we’ve forced kids to run in long enough.
Christine,
He will definitely see your comment.
Christine,
Jay Mathews read your comment and responded to me:
The writer makes excellent points, but I think our differences are summed up by the line “Most selective colleges we researched do not take AP credits outright.” That’s right, but the entire comment is focused on students like that, in the top ten percent. That is not why I started to write about AP with my Escalante book and not why I wrote about IB in “Supertest” and not why I wrote about how to prepare average kids for AP and IB in “Question Everything.”
The vast majority of high school students do not attend selective colleges. They attend state colleges and community colleges, and for those students AP and IB are by far the best preparation available. If the writer knows of something better, let me know, but it has to be something readily available in Detroit, Brownsville, East LA and Miami, not Scarsdale, Palos Verdes, Bethesda and Palo Alto.
I bet the writer has talked to dedicated AP and IB teachers in communities far from our wealth centers. They are the people who have inspired most of my work. The College Board may be a big organization with lots of flaws, like the SAT, but AP turned out to be a great tool for average and below average communities. People who judge it based on what they see in affluent schools are missing something. (Although I would argue that even there, AP and IB, are usually better taught than the alternatives, particularly if they let in every student who wants a challenge.)—Jay Mathews
I am not aware that AP is marketed any differently in high schools who typically send large numbers of students to more selective colleges. Schools are ranked by them no matter where they are and no matter what colleges their students attend. Nobody tells you upfront that the colleges you are planning to attend will pay little attention to AP scores.
Christine, You ask why bother to take AP or IB courses?
Rigor and to aquire the skills needed for succeeding in college level classes.
Perhaps, as Jay said, that applies to students who come from high schools in less affluent communities. My experience both as a parent and a teacher makes me question how many students woud actually credit their college success to the rigor and the skills acquired by taking APs. Most students who took APs already had the necessary skills. IB was not offered in either the affluent community in which I live or the low income community in which I taught.
Well for students who grew up under the umbrella of Special Ed, as well as those in general education who were not taking Honors level classes or higher, in many districts I am familiar with in the upstate NY public schools, I can tell you most parents will say otherwise.
I can appreciate Carol Burris’s critique of Jay Mathews Challenge Index. But her criticism falls way short. Advanced Placement is NOT what Mathews – or Burris – thinks it is. And Burris is wrong; Mathews should NOT make two Challenge Index lists; he should make none at all.
The Challenge Index has always been a phony list that doesn’t do much except to laud AP courses and tests. The Index is based on Jay Mathews’ dubious assumption that AP is inherently “better” than other high school classes in which students are encouraged and taught to think critically. The research on AP makes clear that it is more hype than anything else.
Let’s examine only a few of the ludicrous statements that Mathews makes, and then dig into the research on AP.
Mathews: “AP courses mimic introductory college courses in state universities. The final exams are written and graded by outside experts and thus are immune to the tendency in high schools to go easy on students…”
Oh, dear God. The truth is that AP courses do not come close to replicating college courses.
As one student remarked, after taking the World History AP test, “dear jesus… I had hoped to never see “DBQ” ever again, after AP world history… so much hate… so much hate.” And another added, “I was pretty fond of the DBQ’s, actually, because you didn’t really have to know anything about the subject, you could just make it all up after reading the documents.” Another AP student related how “high achievers” in his school approached AP tests:
The “outside experts” Mathews cites as the “graders” of AP exams are mostly high school AP teachers who read (rapidly) AP essays (the rest of the exams are machine scored). One of these “experts” discussed the types of essays he saw:
“I read AP exams in the past. Most memorable was an exam book with $5 taped to the page inside and the essay just said ‘please, have mercy.’ But I also got an angry breakup letter, a drawing of some astronauts, all kinds of random stuff. I can’t really remember it all… I read so many essays in such compressed time periods that it all blurs together when I try to remember.”
Many colleges and universities are finding that AP courses offer relatively little to students. A very large number of colleges restrict AP credit to test scores of only a 4 or 5 (for example, Baylor, Boston University, Chicago, Colorado, Northwestern,William and Mary). And many limit the number of credits that can be used. More (Boston College, MIT, Michigan, Washington University) are limiting scores to 5s or or not allowing AP credit whatsoever. As one AP test grader said, “the scores signify less and less. Anything under a 5 should be suspect. I wouldn’t give anyone college credit for an AP test grade if I had anything to do with it.”
Here’s another Mathews doozy:
“The growth of AP…participation has also been fueled by selective college admissions offices using that as a measure of a student’s readiness for higher education.”
Yet, more and more colleges are finding that AP is – in fact – NOT a measure of much of anything, except of a student’s desire to ge into the college of his or her choice, It’s a game.
The primary reason many students take AP is not to “learn” or to gain “college readiness,” but to game the admissions process. Students feel like they have to put AP on their transcripts or they won’t get into the college of their choice. It’s all about “looking good,” and boosting the grade point average.
One very honest AP teacher wrote recently that “Our district has told the counselors to promote the AP program with scare tactics that they will not get into the college of their choice, the district has incentivized taking the courses with up to 4.5 G.P.A. credit.” Yet, research chows clearly that the more weight AP courses are given, the less predictive power the weighted GPA has for college success. Dumb, dumb, dumb, dumb, dumb.
And, Mathews gives us this pearl:
“The National Math and Science Initiative has spent more than $200 million encouraging schools to add AP courses and motivate students to pass them, while training more teachers.”
The National Math and Science Initiative (NMSI) pushes for more STEM training in public schools, and purports “to bring best practices” to classrooms in order to “to reverse the recent decline in U.S. students’ math and science educational achievement.” NMSI claims to be the only group in the U.S. that “rigorously researched and replicated math and science programs that have produced immediate and sustainable results.” Three big problems present themselves though. First, there is no STEM crisis in the U.S., far from it. Second, there has been no “recent decline” in math and science achievement, And third, the “proven program” cited by NMSI most often is Advanced Placement, and the research just doesn’t support the claim.
Researchers Lindsay Lowell and Hall Salzman note in “Into the Eye of the Storm, that there is no STEM crisis in the U.S. They point out that “the math and science performance of high school graduates is not declining and show improvement for some grades and demographic groups.” They add that on the Trends in International Mathematics and Science Study (TIMSS) “there has been no decline and even some improvement” in U.S. student scores. And they add this:
“The weight of the evidence surely indicates not decline but rather indicates on ongoing educational improvement for U.S. students. This improvement is not only in math and science but in all subjects tested and, importantly, occurs at the same time as a greater and more diverse proportion of the population is remaining in school…The notion that the United States trails the world in educational performance misrepresents the actual test results and reaches conclusions that are quite unfounded.”
Moreover, Lowell and Salzman make clear that there is no shortage of STEM workers. in fact, “the U.S. has been graduating more S&E [science and engineering] students than there have been S&E jobs” for quite a while, and “addressing the presumed labor market problems through a broad-based focus on the education system seems a misplaced effort.” They add that “policy proposals that call for more math and science education, aimed at increasing the number of scientists and engineers, do not square with the education performance and employment data.”
Beryl Lieff Benderly wrote this stunning statement recently in the Columbia Journalism Review:
“Leading experts on the STEM workforce, have said for years that the US produces ample numbers of excellent science students…according to the National Science Board’s authoritative publication Science and Engineering Indicators 2008, the country turns out three times as many STEM degrees as the economy can absorb into jobs related to their majors.”
This STEM focus may be trendy, but it is based on a fallacy. It’s a myth.
The National Math and Science Initiative is funded by the Gates and Dell Foundations (which seem to have a distaste for public schools), by defense contractors like Northrop Grumman and Lockheed Martin (which have laid off STEM workers), by ExxonMobil (which has funded and disseminated disinformation on climate change), by JP Morgan Chase (which sold toxic securities, defrauded the public, and helped to cause the Great Recession), by Boeing (which is subsidized by US taxpayers, pays a lower tax rate than most American workers, and has laid off thousands of STEM employees), and by the College Board, maker of the PSAT, SAT and AP courses and tests.
Meanwhile, some of the world’s biggest banks and trading companies gamed a “market” of some nearly $400 trillion interest-rate trades, and not in favor of the public. And, more recent disclosures reveal that traders and bankers have rigged the foreign exchange (FX) market, one that involves daily transactions of nearly $ 5 trillion, which is “the biggest in the financial system.” As one analyst noted, this is “the anchor of our entire economic system. Any rigging of the price mechanism leads to a misallocation of capital and is extremely costly to society.”
We have a person in the White House who ran a decidedly racist, xenophobic, and misogynist campaign, and who was helped into office by Russian intelligence agencies who hacked, leaked and falsified documents to harm his opponent, and who has fired the FBI director in an overt attempt to quash investigations into subversion of democracy. Yet, Jay is still pumping out drivel about America’s “best” schools based on a deeply flawed Index that he cooked up and that has no basis in research.
Sadly, Burris seems to support his mania over AP.
The Challenge Index is based mostly on the number of AP tests that a school gives. And the research on AP finds it to be more hype than useful educational tool. It’s past time to let it go. But will educators release it?
Democracy,
Carol Burris asked me to post this response to your comment:
Democracy,
I do not share Mathews’ mania for AP. I suggest you read this: https://www.washingtonpost.com/news/answer-sheet/wp/2017/05/21/the-problems-with-2017-national-rankings-of-americas-high-schools/?utm_term=.52ab2e66d1f8 where I critique the list.
In fact, I started the Schools of Opportunity Program with Kevin Welner several years ago in order to combat Jay’s list and the US News list.
I grew to deeply dislike AP, and worked to eliminate all of AP in my high school. I did so with the exception of Calc and Stats before I retired. I do believe in IB. It is a rich, accessible curriculum with embedded assessments that make sense that are not multiple choice.
Putting that aside, some charter schools are less than honest in their reporting (I am taking a look at the entire list) and are using the list to promote their schools. That must be exposed. They are also not serving the interests of their students well. Please again, read this: https://www.washingtonpost.com/news/answer-sheet/wp/2017/05/21/the-problems-with-2017-national-rankings-of-americas-high-schools/?utm_term=.52ab2e66d1f8
Whether you or I like AP, it is not going away anytime soon. What Diane posted was a tiny slice of a dialogue with Mathews that I shared with her in an email. It has been part of an ongoing conversation that I am having with him. He will not abandon his list, but if I can convince him to not let charters use his list to pump their numbers by giving kids a ridiculous amount of tests and also reward their attrition, that is better for all of the kids who presently attend charters.
I do not believe in the charter school philosophy. I do believe in the kids who attend them, and if we can make their experiences better, I always think that is good thing.
“Whether you or I like AP, it is not going away anytime soon.”
God, that’s the crap line I used to hear from my adminimals when I told them that I refused to teach AP Spanish. Yes, I was “certified” (I think they put a purple round stamp on my arse) to teach AP Spanish. A complete waste of time, effort and more importantly, the students’ parents’ money.
After shilling for Michele Rhee, KIPP and so-called education reform for many years, no one should believe a word excreted from Jay Mathews word processor.