Mercedes Schneider writes here about the strange events surrounding Manuel Alfaro, an ex-College Board employee who has been complaining loudly about the defects of the redesigned SAT.
His home was raided by the FBI, very likely searching to find out if he was the one who released 400 test items to the media.
The day after the raid, he wrote a long commentary about the flaws of the new SAT.
Diane Ravitch's blog 
We should stay alert here: reformers now using the FBI to intimidate opponents. One would think the full resources of the FBI would go towards defending the “homeland.” Or is rooting out opponents of Coleman now part of the defense of the homeland?
Mr. Alfaro should be a “hero” of this blog.
Great idea, Steve!
Agree with your larger point, while also hoping that the fallout from the whistleblowing Manuel Alfaro will leave David Coleman without a job and the SAT test and “brand” with less support.
“. . . he wrote a long commentary about the flaws of the new SAT.”
Noel Wilson has written an even longer “commentary” about the flaws of all educational standardized tests. New SAT, Old SAT, ACT, NAEP, PISSA, whichever you choose are all fundamentally, onto-epistemologically flawed for they attempt to “measure” the “nonobservable”, i.e., latent characteristics, for which there is no standard unit of measure, no exemplar of that unit and no measuring device calibrated against that non-existent standard unit of measurement of the teaching and learning process.
To begin to understand all of those errors, falsehoods and pyschometric fudges (which is to what Alfaro is pointing out) read and comprehend Wilson’s never refuted nor rebutted “Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine.
1. A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
2. A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
3. Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
4. Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other words all the logical errors involved in the process render any conclusions invalid.
5. The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
6. Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
7. And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self-evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
Duane:
You and I and many others who reject these kinds of tests can still recognize, applaud, and support people who call out fraud in the testing industry, can’t we? I think we can forgive Mr. Alfaro his Foucaultian error for his service here. Don’t you?
If I wanted to detract from Alfaro’s work I would have said so. I applaud his work and would place him on Diane’s honor roll. (not into hero designations). I’m fairly direct when commenting and don’t “speak diplomatically” when I talk of those who implement nefarious and harmful educational malpractices. I’m just adding to what Alfaro wrote, nothing more or less.
Why do I feel like we’ve been annexed by Putin’s Russia?
Because you have, in a sense.
That’s why.
Oh, the good ol USA learned a lot from the Soviets and East Germans in regards to controlling the population. What we didn’t learn was that those regimes of control ultimately fail.
Chilling, Duane, that the FBI is used to strong arm and intimidate a citizen, not for criminal activity, but to further a corporate and ideological agenda. We’ve turned America over to corporations and this is the result.
Colleges seem to already know about the flawed SAT. In college visits this summer with my son, we went to 4 different engineering programs. 2 public and 2 private. The message we got from these visits was that they will look at any test scores, but they are much more “confident” in ACT as opposed to SAT.
I will add that not all of them even wanted the scores.
That is encouraging. Too many colleges we visit use SAT/ACT scores to negotiate down the “retail” price. For a middle class family without a lot of AP courses, college is still unaffordable.
As more colleges have entered the Higher Ed world and competition for student money has become part of the Higher Ed experience, more and more colleges are beginning to see that looking at the old-school idea of using standardized “culture-specific” test scores as their means to find students is much too limiting.
Barbara Mikulski has alerted the Consumer Protection Bureau to Alfaro’s claims. I have written to my senator, Elizabeth Warren.
Outrageous that taxpayer money is used on an FBI raid (!) for David Coleman’s bureaucratic pissing contest.
Hey, gotta protect them corporate persons, donchaknow!
Mathman: It’s more due to the fact that colleges now routinely use their own placement testing in order to obtain similar information that was historically provided by the SAT testing. They also know that the majority of these young adults will be struggling with numeracy & literacy skills (both reading & writing) because they were passed through the primary and secondary educational systems without the requisite skills solidly in place. Besides, they are trying to remain in business and therefore if the skills of their pool of applicants have been dwindling, then, of course, they are no longer going to “require” SAT scores, since they already know that the majority of students skills will need to be scaffolded at the college level.
“. . . without the requisite skills solidly in place.”
Or is it without the requisite knowledge solidly in place?
Coleman is heading an organization which is quickly making itself obsolete. With the new changes resulting in an even greater inequitable assessment (vs the SAT of old), more and more colleges and universities will either rely on the ACT or do away with the entire College Board requirement.
It just seems that whatever Coleman touches falls apart. He’s becoming the kiss of death (a title which is well deserved).
Coleman – retire and get out of your own way. You are a walking disaster who has done so much harm I’m surprised you are willing to show your face in public.
Here’s the thing: Both the ACT and the SAT do not predict well success in college. Far too many teachers and counselors (and parents) have fallen into the trap of believing the hype about both tests. So too do they buy into the idea the Advanced Placement courses – produced by the College Board – are better than regular high school courses that focus on critical thinking.
The problem extends far beyond high school. Both testing organizations produce “college and career readiness” tests that are used as early as sixth grade. Newspapers like Education Week – targeted to “teacher-leaders” – publish articles with mind-numbing and inaccurate conclusions like this:
“More students than ever are taking the SAT and the ACT, yet recent reports on performance for the class of 2015 suggest that most of them are ill-prepared for the academic rigor of college.” Community colleges routinely use placement tests – COMPASS (by ACT, Inc.) and ACCUPLACER (the College Board) – to determine whether or not students are ready for college work. Even though the tests are woefully poor, the research underscores that they shouldn’t be used for that purpose, and here’s their practical effect on students and the community college system: “59 percent were referred to developmental math and 33 percent were referred to developmental English (Bailey, Jeong, & Cho, 2010). Students must pay tuition for remedial courses, but the credits they earn do not count toward graduation requirements. The cost to schools of providing this remedial instruction has been estimated at $1 billion or more.”
How dumb is this?
The SAT is, in essence, a proxy for family income. And it doesn’t do much. John Katzman, founder of the Princeton Review, a test-prep company that specializes in helping students prepare for and “beat” the SAT, said this:
“The SAT is a scam. It has been around for 50 years. It has never measured anything. And it continues to measure nothing.”
The SAT predicts about 14 percent of the variance in freshman-year college grades (the ACT does slightly better), and after that nothing. College enrollment experts say that shoe size would work as well.
See: http://www.theatlantic.com/magazine/archive/2005/11/the-best-class-money-can-buy/4307/
College administrators are influenced by rankings like those produced by US News & World Report. They use ACT and SAT scores – an important component of the rankings – to help them move up. Both the ACT and College Board “sell hundreds of thousands of student profiles to schools; they also offer software and consulting services that can be used to set crude wealth and test-score cutoffs, to target or eliminate students before they apply.”
The ACT and the SAT are used for the purpose of “financial-aid leveraging.” Instead of using a $20,000 scholarship for one needy student, schools can break that amount into four $5,000 grants for wealthier students who score higher, who will pay the rest of the tuition ($15,000 a year) and who will bring the school more cash and “will improve the school’s profile and thus its desirability.” In other words, affluent students who don’t need financial aid receive it, and low-income students get hosed so “elite” schools can boost their own profiles (and some administrators can get cash bonuses).
The manufacturers of the ACT and SAT claim that they are very good predictors of college success. But it simply isn’t true.
The authors of a study in Ohio found the ACT predicted only an average of 11 percent of the variance in freshman-year Grade Point Average. Here is what the authors say about the ACT in their concluding remarks:
“…why, in the competitive college admissions market, admission officers have not already discovered the shortcomings of the ACT composite score and reduced the weight they put on the Reading and Science components. The answer is not clear. Personal conversations suggest that most admission officers are simply unaware of the difference in predictive validity across the tests. They have trusted ACT Inc. to design a valid exam and never took the time (or had the resources) to analyze the predictive power of its various components. An alternative explanation is that schools have a strong incentive – perhaps due to highly publicized external rankings such as those compiled by U.S. News & World Report, which incorporate students’ entrance exam scores – to admit students with a high ACT composite score, even if this score turns out to be unhelpful.”
Manuel Alfaro is helping to unmask the SAT, to make clear what it is (a poor test) and what it is not. But this is not really a new story.
It is, however, a story to which more people should be paying attention.
“And it continues to measure nothing.”
Even one of the country’s most ardent proponents of standardized testing Richard Phelps agrees:
“Richard Phelps, a staunch standardized test proponent (he has written at least two books defending the standardized testing malpractices) in the introduction to “Correcting Fallacies About Educational and Psychological Testing” unwittingly lets the cat out of the bag with this statement (notice how he is trying to assert by proximity that educational standardized testing and the testing done by engineers are basically the same, in other words a “truly scientific endeavor”):
“Physical tests, such as those conducted by engineers, can be standardized, of course [why of course of course], but in this volume , we focus on the measurement of latent (i.e., nonobservable) mental, and not physical, traits.” [my addition]
Now since there is no agreement on a standard unit of learning and there is no measuring device calibrated against said non-existent standard unit, how is it possible to “measure the nonobservable” which is what all this standardized testing insanity, truly insanity if you think about it, is about???
So much harm to so many students is caused by the educational malpractices that are standards and testing or as Phelps contends in “measuring the nonobservable”.
How insane is all of this???
Utterly beyond my comprehension!!!
Duane, just to (try to) clarify: when Phelps says “…tests can be standardized … of course…” I think he’s referring to the technical definition of score standardization. That’s when you recalibrate scores so the set has a mean of zero and standard deviation of one, aka “z-scores”. It’s a mathematical manipulation and it can be done “of course”.
Standardized tests OTOH are a whole ‘nother kettle of beeswax. That refers to the effort to test all kids against a common standard. I think post-facto score standardization goes on there too, when the scores are “normed”. But clearly SAT scores don’t look like zscores. However they may be jiggered to be “normally” distributed.
But what we object to in standardized tests is not the standardization” of scores, but the conceit that learning, and the testing of what is learned, can be “standardized”. There are so many locuses of disconnect along that route; the faux-objective effort to explain complex parameters simplistically, and reductively, is really suspect.
What’s worse is the expectation that everyone will “pass” – forgetting the scientific notion of the bell curve.
Then there’s the idea that children will show more than ones year’s growth each school year. If they are functioning as a 3, they will most likely stay a 3 (or a 1, 2, or 4). My principal thought that the teachers should be able to move the kids up a notch, which is possible (although not likely), however, that’s assuming the testing and scoring stays the same each year. When exams are unfair and the scores are predetermined, the entire idea of testing is erroneous.
As I’ve said before, my grand daughter went from a 3 in third grade to a 2 in fourth grade to a 1 in fifth grade and then a 0 in sixth (since she opted out), I guess she got stupider each year since not only did she show no progress but she actually “deproved”.
One word: WOW!
My district just bought McGraw-Hill online textbooks for me to use. Very Common Core. Test preppy. Close reading of very short, info text, blah blah blah. The first lesson: how to send a tweet. I kid you not.
My school had to choose between McGraw-Hill and College Board for English textbooks. I still think we made the right choice.
Not College Board!
Sorry, that’s #not college board in 140 characters or less.
But why the FBI? David Coleman’s disasters are international. Shouldn’t they have sent the CIA?
Reblogged this on Matthews' Blog.
Reblogged this on An Alchemist's Journey…. and commented:
We need to take back our planet before the powers that be ruin it for us ALL!