A regular reader uses the pseudonym. He/she posted this comment about high school rankings:

“It isn’t just the US News & World Report school rankings that are dubious. It’s also Jay Mathews Challenge Index rankings at The Washington Post, which have been discussed on this blog.

“But there’s another set of rankings that are equally suspect. And no one is saying anything much about them. These are the school rankings produced by an outfit called Niche. And they are not very good.

“First, some history.

“Niche is a private, for-profit company. It began as College Prowler, a college guidebook company. Some questioned its college rankings and reviews, and its less-then-ethical practices. Some higher education experts criticized the “College Prowler scandal, in which the purveyor of college guides was caught impersonating both students and colleges on Facebook in order to mine data and drive traffic to its website.” More information on that scandal can be found here:

https://www.insidehighered.com/news/2008/12/22/frenemies-facebook

“The Chronicle of Higher Education noted that “College Prowler had formed a partnership with [another] company to ‘colonize’ Facebook groups for marketing purposes.”

“In 2015, the architectural engineering company SmithGroupJJR, one of the top such firms in the country, noted in its Perspectives blog that the Niche college rankings were of “questionable” value and – importantly – observed that Niche is working toward a future of trying to get its rankings used as ” a viable future replacement for SAT/ACT scores…”

“Niche now ranks public schools too, in each state and across states. It describes its methodology for “Best Academics” as “a comprehensive assessment of the quality of the academics at public school districts in the United States.”

“But it isn’t really that at all.

“Nearly a third of that assessment relies on dubious data. For “Best Academics,” Niche allots 10 percent to the number of students taking an Advanced Placement (AP) course, 10 percent for the percentage of students who “pass” an AP course, and 10 percent for SAT/ACT composite score. The Niche ‘best academics” assessment jumps to about fifty percent when student “interest” in an “elite” college is added in. Niche also uses state testing data.

“Research shows clearly that AP is not what people think it is. For example, a 2002 National Research Council comprehensive study of AP math and science courses and tests found they were “a mile wide and an inch deep” and did not conform to research-based principles of learning. A “3” on an AP test is considered a “passing” score, but it equates to a “C” in a college survey course, and most colleges – especially the “elite ones” – do not award any credit for a “3.” Students freely admit that they take AP course primarily to “look good” rather than to enrich their learning.

“As I’ve noted before, AP may work well for some students, especially those who are already “college-bound to begin with” (Klopfenstein and Thomas, 2010). As Geiser (2007) notes, “systematic differences in student motivation, academic preparation, family background and high-school quality account for much of the observed difference in college outcomes between AP and non-AP students.” College Board-funded studies do not control well for these student characteristics (even the College Board concedes that “interest and motivation” are keys to “success in any course”). Klopfenstein and Thomas (2010) find that when these demographic characteristics are controlled for, the claims made for AP disappear.

“A newer (2013) study from Stanford notes that “increasingly, universities seem
to be moving away from awarding credit for AP courses.” The study pointed out that “the impact of the AP program on various measures of college success was found to be negligible.” And it adds this: “definitive claims about the AP program and its impact on students and schools are difficult to substantiate.”

“But Niche has glommed onto the AP myth. The SAT and ACT too.

“SAT and ACT scores are little more than proxies for family income. They are not accurate predictors of success in college. College enrollment specialists say that their research finds the SAT predicts between 3 and 14 percent of the variance in freshman-year college grades, and after that nothing (the ACT is only marginally better). As the head of one college enrollment consulting company commented, “I might as well measure their shoe size.” Moreover, colleges – especially “elite” ones – use SAT and ACT scores to enhance their own prestige and to exclude poorer students from admissions.

“The Niche methodology for “best schools” also utilizes survey responses. Niche says that there must be a minimum of “11 unique respondents required at each district.” If that is for the entire district, then it’s a pretty doggone small sample. Moreover, the responses on which the rankings are made must come from “registered users.”

“All of this raises multiple questions. Why is Niche using suspect data like AP and SAT/ACT scores to rank schools? How – exactly – does one becomes a “registered user”? What are the demographics of Niche “registered users”? What does a sample “survey” that Niche gives to “registered users” look like?

“Guess what? If you asked Niche – even if you asked multiple times – you’d not get any answers.

“Interestingly, if you look at the Niche “best schools” rankings, all of them are interlinked with Realtor.com.

“Perhaps even more interesting, and bizarre, is that one central Virginia school division — a school division that touts AP courses and SAT scores, and that has gone all-in on the STEM fallacy, and that bills itself as “innovative” and “cutting edge” — has adopted the Niche rankings as the basis for a “market” that determines how it pays its teachers.

“The founder of College Prowler/Niche says this about Niche evolved out of College Prowler:

“Only a couple million people a year choose colleges. It’s not a market like Facebook…we needed more visitors and more dollars per visitor. How much traffic you have and how well you monetize this traffic is at the core of everything…So we rebranded from College Prowler to Niche…to a much larger market…We wanted to be a very big company, and now that’s what we’re on the path to do.”

“So, Niche, is “on the path” to market its mostly made-up school rankings to suckers, make big money doing it, and it has the help of a public school system that has already bought into all the educational goofiness that’s out there and yet has the gall to call itself “innovative.”

“I cannot help but to recall the line from Forrest Gump: “Stupid is as stupid does.””