Jay Mathews writes about education for the Washington Post. Although he and I don’t always agree (he is a huge admirer of KIPP and wrote a book about it), I have always found him to be extremely congenial. He recently sent me the following note, asking for a correction, and I asked and received his permission to post it in full. I don’t recall whether Tom or I wrote the sentence he quotes, but I want to set the record straight no matter who wrote it.
He wrote:
Hi Diane—-I was having fun reading Tom Ultican’s blog, then found something you wrote last year that needs a slight correction. You said:
Mathew’s methodology has now become the US News and World report ranking of “the best high schools” in the nation.
Many people have the same impression. It is hard to keep this stuff straight. My methodology has NOT become the US News high school list. I started mine in 1998. It ran in Newsweek for many years. US News started theirs in 2007. Their method is quite complex, with an emphasis on average test scores and extra points for low income schools that show test score gains. They also include something similar to my counting participation in AP and IB tests, but it is a small part of their method. I like the US News guys but complain that any method that includes test scores becomes misleading. I have noted frequently that their number one school, Thomas Jefferson, in a system that is supposed to detect the best low income schools, has only 2 percent of its students from low income families.
My Challenge Index list is now on my website, jaymathewschallengeindex.com. I exclude TJ and all other public schools that have average SAT or ACT scores above the highest average for any neighborhood high school in the country.
I hope you are having a splendid summer. —jay
Diane Ravitch's blog 
Where is Matthews’ latest book where he shills for the charter industry by proving they are cheaper and better. He was cozy with the IDEA founders— hasn’t said a word about their gross malfeasance. Texas charters do not use less money. Less funding is used for instruction because of bloated administration, but only 50% of state tax dollars are in the classroom. As far as using AP as in indicator of best schools—not valid.
Charter School Management 101: You get x amount per student from the state. Your finances are not transparent. Anything you don’t spend on kids and their instruction is profit. Game on.
Any system of rating and ranking relies on how much weight each category is given in the formula for calculating results. Algorithms are only as valid as the humans that assign value to the various categories, and we know they can be misleading or inaccurate. How many teachers were sent to the chopping block from misleading VAM scores? How many misleading rankings have we seen from “months of learning loss or gain” in some charter schools’ claims?
In April Wallet Hub rated “The Best Public High Schools in NYS. While they explain how they arrived at their decision by using assigned weight to various categories, the result depends on the perceptions of those assigning value to the various categories. No school is perfect, but at least most public high schools provide options and opportunity for a variety of students based on individual needs and interests.https://wallethub.com/edu/e/best-public-high-schools-in-new-york-state/88443
I basically never agree with Jay, but he is amazingly nice. He has staked out his position on education “reform” scams (his position is almost full-out pro*) but manages to float serenely above the fray. (Disclosure that he quoted me in his KIPP book because I did early number-crunching on their eye-popping attrition.) *Except that Jay’s wife was the editor on a project that found a high level of test-score juking in took-the-“reform”-money-and-vanished Michelle Rhee’s Washington, D.C., schools, so he has refrained from puffing Rhee.
“He has staked out his position on education “reform” scams (his position is almost full-out pro*)”
Please clarify what you mean– 1) He is PRO what he considers to be legitimate reforms while others regard them as scams, OR 3)He is PRO exposing the scams because they damage what he regards as true reform efforts?
Everything coming from what has been known as the “education reform movement”’over the past 25-30 years is a scam.
So Jay believes that “Everything coming from what has been known as the “education reform movement”’over the past 25-30 years is a scam.” ?
The Summit charter chain in California also claims its students have to pass X number of AP exams, but parents with kids at those schools have said “huh?” — they haven’t heard of that. So some cheating, juking, BSing or something seems to be going on — what a surprise from the charter sector — that’s the way they roll.
No, that’s my view. Jay is a cheerleader for the scams except for the Michelle Rhee shenanigans that his wife helped expose.
Here’s the difference between a scholar like Diane Ravitch and an [pejorative withheld] like Jabba the Trump: On the rare occasion when Dr. Ravitch makes or might have made an error, she admits it publicly. When Stable Genius [ROFLMAO] Trump makes an error, which he generally does at least once but more typically many times whenever he opens his maw, he says he was joking and that the reports on what he clearly did say and intend are “fake news”.
Why does California fare so poorly in Jay’s rankings?
This is interesting. If I had quoted you, there would be a hyper-link. So, I started doing a word searches of my blog and could not find the quote. It sounds like something I would have written and his explanation might have technically pointed to you or I making a slight error but not really a misleading statement. I suspect our friend Jay who likes publicity is more interested in that than correcting an error.
“My methodology has NOT become the US News high school list.”
SWELL!
Yet the high horsemen, annointed from the oil of concocted notoriety mythology, continue divisional rankism.
It’s past time to get off your high horse and take a walk
in the shoes of those you “think” you’re helping.
“When everbody’s somebody, then no one’s anybody.”
Good points.
Snake oil salespeople act like very nice people. That’s their job. The Challenge Index is pure nonsense with an audience of the gullible. Matthews tells people a school is good because the students took lots of standardized tests. Pathetic.
Yes, it’s like the Mr and Mrs Nice Person facade that Bill and Melinda Gates fooled the media, Presidents and the public with for so many years.
Meanwhile, behind the scenes they were meeting with convicted pedophile Jeffrey Epstein after his conviction for paying a minor for sex, at the same time that they and their foundation were claiming go be “championing” the cause of young women.
And despite Melinda’s recent claims that she disapproved of Bills meetings with Epstein, she nonetheless attended at least one of those meetings in 2013, AFTER Epstein’s jail term. And all these years she said not one word about any of it.
Why didn’t he identify the date of the past, so it would be easy to go back and find out who said what. If he found it, it would have been a simple matter to include that information. I don’t doubt that it was said, but since he included that quote, including from whence it came would have been easy, not that it would or should make any difference in your publishing his correction
This seems like a minor correction, so I would like Jay Matthews to go on record right now — at this blog — that if we find any minor errors in anything he has written for the past 5 years, that Jay Matthews would similarly issue a correction and note that he made a mistake.
If Jay Matthews won’t do that, he is a hypocrite. It doesn’t matter if something he wrote is partially correct if it is not precisely accurate. Does he hold himself to the same standards? Especially when he promotes charter exaggerations of their performances? Remember, if Matthews isn’t 100% accurate, he must issue a correction. Or admit that Diane Ravitch’s integrity is something that he himself does not have.
Challenge made, Mr. Matthews. Will you hold yourself to these standards and issue corrections?
“I exclude Thomas Jefferson and all other public schools that have average SAT or ACT scores above the highest average for any neighborhood high school in the country.”
Will someone please explain the distinction Matthews is making between “Thomas Jefferson and …other [high scoring] public schools” and “NEIGHBORHOOD high schools” ?
How does he decide or determine his cut-off point to be “above the highest average for any NEIGHBORHOOD high school” ? Are there no NEIGHBORHOOD high schools with “high” scores?
He also doesn’t exclude the BASIS charters, run by a for-profit management company owned by the same people who run the charters.
I wonder if there is a higher rate of poverty at Thomas Jefferson than at BASIS! Jay Matthews just puts “N/A” in the column that lists “free lunch” for all the BASIS “public charters” in Arizona.
I wrote here a couple of years ago that the Blocks, who own the BASIS chain, bought an apartment in NYC for $8 million.
Here’s a clear picture of the problems with charter schools — this is specifically about California but generally applies overall: https://teachingmalinche.com/2018/04/29/whats-wrong-with-charter-schools-the-picture-in-california/
Good article – thanks!
When you have a bogus ranking system (sorry for the redundancy), you can make up any rules you like.
Asking how he decides details ” is like asking how a monkey decides which banana it eats first in a bunch.
Can anyone explain why Jay Matthews thinks that BASIS charters in Arizona are outstanding, but not Thomas Jefferson?
Also, Jay Matthews claims that BASIS Charter schools in Arizona are “public charters” but they are for-profit and calling them “public charters” perverts the meaning of the word “public” and presumably Jay Matthews now supports the Republican view that charters should be allowed to make a profit.
Any corrections, Mr. Matthews?
I am somewhat familiar with four AZ charter schools. My incidental knowledge is—
All are open to the public and two of those I know to be operated for profit; don’t know about the other two. I happen to know at least one professional, non-doctrinaire parent in each school that is very happy with the programs.
One of these parents was and is very supportive of the public school his three children were enrolled in–and where I worked–before enrolling them in a charter for high school. His reason: more opportunities, including classes at the local community college with college credits earned.
Another of the charter schools is K-8 with a true emphasis on the arts, which is a strong draw from what I hear anecdotally.
I worked as an arts teacher part-time for two years at one of the for-profit charters after my public school job was cut in half during the Great Recession. My salary–like everyone else’s including the principal’s–was $32,000 (but prorated for me because I was part time) versus $42,000 in the public school. All the parents I specifically asked during those two years (a dozen or so) said they moved their kids to the charter because their academic or personal needs were not being met in the public schools. To my knowledge, no student was every dismissed from that charter for grades or behavior, and the school had an excellent special ed teacher. My classroom was 1/3 occupied by surplus equipment and other items in storage. That was bad, but gave me more room than I had on the cluttered stages-masquerading-as-warehouses in three of the four public schools I taught at during two decades.
When the BASIS school was starting out several years ago, the music teacher told me he was expected to teach band and orchestra students in the same room at the same time, but perhaps that has changed now.
Mark,
Carol Burris studied the demographics of the BASIS charters in AZ, which are usually rated “best in the nation” because of their high test scores. She found that BASIS schools serve a population that is overwhelmingly white and Asian, and serves small numbers of students who are Hispanic, Native American, or African American, and minuscule numbers of students with disabilities. Students who can’t pass a large number of AP exams can’t graduate, and are unlikely to seek admission.
You can Google Carol Burris and BASIS. The chain boasts very high test scores. Their secret sauce: demographics.
She writes:
“Critics of charter schools have long observed the differences in school populations that charters serve, and charter schools counter that that is not by design. A quick look at the demographics of the 18 Arizona BASIS charter schools compared with the demographic profile of all Arizona students in the public and charter systems, however, should give pause that such differences are not accidental. The following enrollment figures are from the 2015-2016 school year.
“The proportional over-enrollment of Asian-American students and under-enrollment of Latino students in BASIS charter schools is startling. But differences in the students served do not end with race and ethnicity.
“In 2015-16, only 1.23 percent of the students at BASIS had a learning disability, as compared to 11.3 percent of students in the state. BASIS schools had no English Language Learners. And in a state in which over 47 percent of all students received free or reduced- priced lunch, BASIS had none. Although BASIS may have some students from qualifying households, it chooses not to participate in the free or reduced-priced lunch program.”
Sorry, I meant this to respond to chartery claims about their students passing AP exams, but it went in the wrong place in the thread, so I’m duplicating it:
The Summit charter chain in California also claims its students have to pass X number of AP exams, but parents with kids at those schools have said “huh?” — they haven’t heard of that. So some cheating, juking, BSing or something seems to be going on — what a surprise from the charter sector — that’s the way they roll.
BASIS Charter Schools
The basis of success
Selection at its best
Cuz Darwin’s ” fittest” cases
Got nothin on the BASIS
The basis of success?
The score on standard test
To just accept the best
And winnow out the rest
“the music teacher told me he was expected to teach band and orchestra students in the same room at the same time”
“The Leader of the Borchestra”
I’m leader of the “borchestra”
The cello-marching orchestra
With Hulks a’toting basses
To march around the bases
SomeDAM, how about this for a fifth line–
Strictly avant-guard at BASIS!
Deming: “Plot the data [on a control chart aka process behavior chart].”
Okay, in the interest of improving learning and decision making, let’s plot the Jay Matthews Challenge Index, 2020 Top 300 Schools Index on a control chart (simulating time-ordered production of the indices, since the indices are attribute data).
The chart says…
17 of the 300 (5.67 percent) school indices, listed below, are detectably different from all the others for the better; they all exceed the 12.2691 upper natural limit of variation, with some doing so to greater extents than others.
* 4 of the 17 represent public schools
* 0 of the 17 represent private schools
* 13 of the 17 represent charter schools
** 5 BASIS charter schools
** 6 IDEA charter schools
** 2 other charter schools
And the chart says…
283 of the 300 (94.33 percent) school indices are not detectably different.
* The 283 school indices are the same, within limits
* The 283 school indices vary between the natural limits of 1.2432 and 12.2691
* 191 of the 283 represent public schools
* 39 of the 283 represent private schools
* 53 of the 283 represent charter schools
** 0 BASIS charter schools
** 3 Uncommon charter schools
** 3 YES Prep charter schools
** 5 IDEA charter schools
** 6 KIPP charter schools
** 7 Uplift charter schools
** 29 other charter schools
Therefore…
The chart begs the question: What’s special or unusual or uncommon about the schools the 17 detectably different school indices represent?
Does Jay Matthews know? Can he explain?
Let it be his “challenge” to do so, here.
Any explanations in terms of the school indices themselves will not contribute to improving learning and decision making.
<<<>>>
The 17 detectably different indices and their schools are:
12.288
BASIS Phoenix – C
Phoenix AZ
20.069
Signature – C
Evansville IN
17.194
Mickey Leland College Prep Academy for Young Men
Houston TX
12.685
IDEA Edinburg – C
Edinburg TX
14.548
BASIS Peoria – C
Peoria AZ
14.000
IDEA San Juan – C
San Juan TX
15.590
IDEA Frontier – C
Brownsville TX
13.922
IDEA Alamo – C
Alamo TX
14.219
BASIS Flagstaff – C
Flagstaff AZ
12.404
Corbett
Corbett OR
14.784
Stanton College Prep
Jacksonville FL
15.908
BASIS Tucson North – C
Tucson AZ
12.735
Uplift North Hills Prep – C
Irving TX
12.761
BASIS Oro Valley – C
Oro Valley AZ
12.857
IDEA Mission – C
Mission TX
13.979
Carnegie Vanguard
Houston TX
13.785
IDEA Pharr – C
Pharr TX
<<<>>>
This is one case where one can completely miss the forest for the trees.
A ranking system is only as good as the basis for the ranking.
Mathews’ challenge ranking is based on the following
”
We take the total number of Advanced Placement, International Baccalaureate and Advanced International Certificate of Education tests given at a school each year and divide by the number of seniors who graduated in May or June.”
That’s it!!??
First, the assumption is clearly that those specific types of courses listed are the only possible “challenging” ones. (If a school offered none of the specific types of courses that Mathews uses, it would not even merit a ranking on his scheme, regardless of whether they might be just as or even more challenging than the types of courses Mathews “accepts”)
Second, it doesn’t even matter how well the students who took the requisite types of courses did in those courses or even how well they did on the test at the end, so long as they took the test.
And this is what is used to rank the supposedly “most challenging high schools”?
Really?
This may well be the stupidest school ranking system on the planet and I’d have to say that trying to figure out why a handful of schools are somehow “different” from the vast majority of the rest under this system is undoubtedly a complete waste of time.
But, without spending any time on it, just off the top of my head, I’d guess that the “different” schools require every student to take a certain number of the types of courses on Matthews list and take the test (or maybe just require that they just take a certain number of the end tests, whether they took a course or not, since it does not even matter how well they do on the test)
“This is one case where one can completely miss the forest for the trees.”
When one cannot see and understand the forest (the bigger, living, dynamic system), and when one cannot see and understand that the forest comprises trees (the smaller, living, dynamic systems) that are interrelated variously within the natural boundary of the forest, and when one cannot see and understand some trees exist at various distances beyond the natural boundary of the forest, perhaps even within other forests, then one just might assume (wrongly) that the extra-forest trees have nothing to do with the forest and, on that basis, erroneously characterize consideration of the extra-forest trees to be a case of completely missing the forest for the trees.
The forest without the trees and the trees without the forest is not the way “the interrelated structure of realty” (MLK Jr) works. Will we ever learn this and get beyond the hackneyed “can’t see the forest for the trees” adage?
“A ranking system is only as good as the basis for the ranking.”
What could possibly be a good basis for ranking trees in a forest in a way that would benefit either or both forest and trees, when every ranking system necessarily puts some manner of competition between the things ranked based on the absurd assumption that the things ranked exist and function independently? If only implied, the purpose of every ranking system is to produce as few winners as possible and as many losers as possible, intentionally, by design. Fine for sport competitions and games and ranked-choice voting. But not fine for ranking dynamic schools composing dynamic school systems, or what are supposed to be schools and school systems.
That said, consider that any measure that is the basis of a ranking system need not be used for the purpose of ranking—not even in sports and games. The only reason ranking happens is because of the way the ranking system designer thinks and knows not what to do with ranking system measurements other than to rank them.
Application of any ranking system measure will necessarily produce measurements that capture and lock in them variation in response to any number and manner of known, knowable, and even unknowable dynamic causes—just like a seismograph does, for example. And sometimes some spectra of the variation may be detectably different from other spectra of the variation—for example, variation due to earthquakes vs. variation due simply to Earth humming along, with nothing unusual or uncommon or special happening.
Ranking is blind to systemic variation. Ranking cannot detect and distinguish between when variation likely means something and when variation likely means nothing, no matter how narrow or how wide the natural range of variation may be. Ranking is good for imposing irrational competition and accountability. Ranking is not good for rational thinking and learning to improve. Ranking is folly. The Jay Matthews Challenge Index is folly. And Deming’s control charts aka process behavior charts are both an antidote and countermeasure for exposing the folly that charter schools are better than public schools.
This is Deming in a conversation with a few hundred educators back around c. 1990: “How many of you have [even] heard of [control charts]? Raise your hands!”
He counts off the few timidly raised hands and later predicts: “Another century will pass before we weave this knowledge [about the power of control charts] in our thinking.”
The Jay Matthews Challenge Index is proving Deming right.
I’ll stop here but could go on. However, let’s note: Understanding “variance” is not a substitute for our needing to understand variation. The former is statistical, the latter empirical.