For years, I have read SAT reports about test results. What you always see is that student scores are correlated with family income. See here for some good graphs.
What I noticed this year for the first time was that the College Board (which administers the SAT) is promoting Khan Academy, a private venture funded by the Gates Foundation. The release says that students who got their SAT prep from Khan Academy got higher test scores. The partnership was revealed earlier this year. The courses are free.
In the past, SAT prep has been characterized by paid courses and the ordering of thick books to help students master word-comparison exercises and memorize the so-called “SAT words.” Official SAT Practice on Khan Academy provides online resources that are tailored to each student to help them pinpoint which skills they need to improve. As they practice over time, their study plan will evolve with them and help them level up to more challenging skills.
When I was in high school, we were told explicitly that coaching did not affect SAT scores. That has since been disproved and the College Board now encourages coaching and tells you where to get it.
Never forget that the College Board is a business, always looking for market share.
Diane Ravitch's blog 
With all those practice tests and coaching of students to take those tests, aren’t they INVALID?
LikeLiked by 1 person
They’d be invalid even without that.
LikeLike
YEP!
LikeLike
I wonder how many teachers have experienced test training meetings where SAT/ACT pushers are brought in to tell teachers how to get their kids to score at higher rates….and then give out clues for teaching kids how to “beat the test.” I remember thinking, as I sat through meetings like this, how far we have come from sanity when district leaders push this as a logical solution for appeasing government test invaders.
LikeLike
What a shame. In the beginning, Khan was doing some really cool stuff. But his organization has sold its soul to the test-em-till-they-bleed Ed Deformer machine, and, in particular, to those parts of the machine financed by you know who, the Dark Lord of Deform himself.
LikeLiked by 1 person
money makes school reform “experts” out of so many who will bow at the feet of those offering it
LikeLiked by 1 person
I agree Khan had good motives and ideas before they were taken over by Bill Gates and Bank of America. It’s really not that great a website though, Khan Academy. I’ve tried it. It’s basically just videos of chalkboard-style diagrams with voiceovers. Could be accomplished with VHS or even reel projectors like the ones used to teach me during the Cold War to duck and cover during nuclear attacks and about venereal disease in gym class when I was a boy. Not that great. Khan was right to say it was just a homework help supplement (before selling out). If Khan Academy correlates with higher SAT scores, it’s more suggestive of the inaccuracy of the SAT than of the test preppy power of Khan.
LikeLiked by 1 person
And the history ones (that’s my specialty) are full of errors.
LikeLike
Kahn Academy provides test prep for the same content it provides for SAT tests. The College Board and Bill Gatee promote that corrupted view of academic achievement.
It is time to send the SAT into the mangrove swamp where alligators romp.
LikeLike
It’s so long past time. The SAT was a scam from Day 1. None of its claims were ever substantiated, and each time that research conclusively proved this to be so, they invented another spurious rationale that then had to be disproved at enormous cost in time and valuable resources.
At first, they called it the Scholastic Aptitude Test and claimed it was a general aptitude test, like an intelligence test. When this was disproved, they started calling it a predictive assessment: The Scholastic Assessment Test. But then research showed that it only predicated success in the first semester of college and wasn’t predictive thereafter AND that teacher grades were better long-term predictors of college success, so they stopped calling it the Scholastic Assessment Test and started calling it just the SAT.
But it was always a scam. A very lucrative scam, but a scam nonetheless. A lot of colleges and universities have wised up and have cut SAT and ACT scores from their admissions criteria.
LikeLike
cs: “Scholastic Assessment Test,” above, s/b “Scholastic Achievement Test.” Sorry, I was writing hastily.
LikeLike
Good synopsis, Bob. It was always a scam, but originally based on another scam, the IQ test of ‘intelligence’.
LikeLiked by 1 person
Yes. A great book on this topic: https://www.amazon.in/Big-Test-History-American-Meritocracy/dp/0374299846
LikeLike
better link. The link above was from the Indian Amazon site.
LikeLike
Before it goes away, watch “The American Experience” on Eugenics. Talks a lot about “IQ” tests and how the government labelled one half (!) of draftees in WWI as “morons.” https://www.pbs.org/video/the-eugenics-crusade-jtaetc/
LikeLiked by 1 person
My favorite book on this topic: The War against the Weak, by Edwin Black. https://www.amazon.com/dp/B00AGIIMWO/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1
LikeLike
Perhaps you know, as well,
black’s IBM and the Holocaust https://www.amazon.com/IBM-Holocaust-Strategic-Alliance-Corporation-Expanded-ebook/dp/B00AGIDA8A/ref=sr_1_22?s=digital-text&ie=UTF8&qid=1540287824&sr=1-22&keywords=america+and+the+nazis
LikeLike
And thanks for the tip about the PBS program!
LikeLike
Watching that documentary right now, “Threatened.” It’s superb!!! Thank you.
LikeLike
The mangrove swamp is a beautiful place. Let us not sully it with the carcass of education reform, which has done all it can to supplant environmental education with the nuts and bolts of chemistry at an age before anyone understands it. I know, Laura, it was just an allusion. Sorry, I am a charter member of the National Association for the Literalization of Metaphorical Expression (NOLME). Strike that. I am the only member.
LikeLike
LMAO!
LikeLike
Roy, I misspoke the song version that was running through my head, Noel Coward’s Mad Dogs and Englishman lyrics…
In the mangrove swamps, Where the python romps There is peace from twelve till two….
LikeLike
Sorry, Laura. Did not get the literary allusion
LikeLike
That’s beautiful, Laura! LMAO! Love it!
LikeLike
Since they are working together and if Khan Academy is providing College Board with the results of the practice work students are doing, what effect could that have on the questions College Board then uses on the SAT? The designing of questions, the outcome of the test scores, if Khan collects specific data on students, how a certain population is doing, etc
LikeLike
Very good questions, Dawn! Revolting, but probably so.
LikeLike
And of course coaching improves SAT scores, and so K-12 principals will force their teachers to force their kids to do this crap.
Now millions of kids are being made to plow through even more test prep, and the opportunity costs of this are enormous.
It’s bad enough that because of Gates and his ilk, in tens of thousands of high schools and middle schools, a THIRD OF THE SCHOOL YEAR is given over to what is EXPLICITLY prep for the exams (pretests, benchmark tests, test prep exercises in lieu of traditional instruction, data chats, and the tests themselves), but even worse, the regular curricula, in English and math, have largely been taken over by test prep activities that slavishly follow the puerile “standards” as if they were a curriculum outline and have replaced traditional activities with ones modeled on the exam questions.
NOW, as if things couldn’t get any worse, in many high schools, students are being pulled out of classes to do Khan Academy SAT prep and/or being assigned Khan SAT PREP in lieu of their traditional homework.
And all the time given to this crap is time stolen from traditional curricula and traditional learning.
Opportunity costs.
What’s become of Sal Khan’s idea due to the corrupting influence of Gates money sickens and saddens me. It’s a sell-out like the one Dar Williams wrote about:
LikeLike
cs: In the piece above, “and the exams themselves” s/b “before the time wasted by the exams themselves)
LikeLike
Yep…my daughter sits in AP English Lit and her teacher gives a packet of work and then excuses herself to go out in the hall to Skype her small child in daycare or to play on her phone. Essays were assigned to 30 students and returned the next day with scores of 0-3… she told them that she didn’t read them because they were for baseline benchmark scores and they will show “growth” on the next essay. It’s the end of October and not a single book has been read. Isn’t an English Lit class supposed to be about the reading of classic literature? I’m disgusted…..and the math area isn’t much better. We do have really high test scores in our district, though! We make it into US News and World Report every single year. High scores or an education?….. you can’t have both and I would much rather my child receive the education.
LikeLiked by 1 person
Thank God there are still parents like you who have a clue! But don’t blame the teacher. She is doubtless doing precisely what her administration has instructed her to do. This is what English class looks like throughout the United States now thanks to Gates and the one he appointed to be the decider for the rest of us, Lord Coleman.
LikeLike
What a lousy teacher! I know not everyone likes AP, but I work my butt off as an AP Human Geography teacher. I read every sentence in every essay. My students have already written three practice essays (plus the other writing they do for my class, which is frequent), and are starting on the third of seven major topic areas. Whatever this “teacher” is doing is NOT teaching.
LikeLike
Thanks for mentioning this, Lisa. We don’t hear enough from parents, and it is them that I consider most accountable. The AP teachers in the school in which I serve are similarly lackluster. Parents are the way out of this mess, so please, I ask you, as a high school teacher myself, to stay engaged in the public school process.
LikeLike
“THIRD OF THE SCHOOL YEAR is given over to what is EXPLICITLY prep for the exams”
Sorry, Bob. The evaluation rubric presses teachers to teach to the test every day. Some schools keep big charts on the wall with children’s names and the skills they did not master on the last “formative assessment”. This labor precludes teachers who read a really exciting narrative and share it with their students. The chart is the guide. Administors from neighboring counties go and see these charts and salivate about the brilliance of such machinations.
Yes, Bob. It is worse than you thought.
LikeLike
That was an estimate. I was thinking that most of the rest of the time is given to curricula that have been test-prep-ized. The meretricious pretend wonks at the Thomas B. Fordham Foundation recently put out a white paper claiming that some very small percentage of time in schools was given to testing. But, ofc, they didn’t count all the crap that you and I are counting, Roy.
LikeLike
And don’t get me started on all the K-12 schools in the country now in which kids can’t get access to their “media centers” because they are being used, all the time, for testing or test prep.
LikeLiked by 1 person
and, in the world of poorest schools, computer labs functioning only partially and sporadically as tech assistant positions are cut from the budget
LikeLiked by 1 person
Yep. In my school, we can’t use the Library for the entirety of 4th term because of testing.
LikeLiked by 1 person
Outrageous.
LikeLike
This is pretty much universal now, I think, in US K-12 schools.
LikeLike
Kahn Academy was great when it started and I used to have my kids use it if they had trouble with math or needed some extra practice. As soon as Common Core came into play, Kahn Academy changed. It became the home of “weird” math! I couldn’t understand the Algebra, my techie husband (he could understand the algebra) didn’t know why Kahn Academy chose a different process (more confusing!!!) to explain the math. We stopped using it. This is no surprise…it keeps the money flowing to all the “right” people. SAT and ACT need to go away for good.
LikeLiked by 1 person
In the early 1980s, the NYS legislature passed a Truth in Testing Act aimed at ETS and the SATs–used by overwhelming numbers of college as the major basis for admissions decisions.
At hearings before the vote, I testified in favor of the heart of the bill–sunshine–disclosure of information about the test items, the differential impact the test had by gender and ethnicity, the statistics the publisher kept hidden from view.
It was interesting to see how Stanley Kaplan–the genius behind test prep courses that high school students took before they faced the high-stakes SAT. How would he address ETS’ claim that the exams were uncoachable?
Kaplan brought his own research to the table and showed that his courses boosted scores considerably, enabling paying customers to have far better chances to get into colleges of their choosing.
In other words, preparation designed to game the exams worked, and this was not an aptitude test of some innate intelligence or cognitive ability among college applicants. (Subsequently, ETS stopped calling it the Scholastic Aptitude Test and moved to the acronym SAT in order to preserve its brand by dropping the misleading “A.”)
It was further illuminating to see how a shrewd entrepreneur, like Kaplan, presented his facts without indignation or any hint that he was offended by a test publisher who, in effect, was attempting to discredit the value of his courses. Instead, I took away from my experience, that people with a vested intere$t in the survival of a crooked business–do not want to rock the boat, if doing so would jeopardize their own fortune.
Truth in testing and transparency are needed more than ever–a need exacerbated by NCLB, Race To The Top, the Common Core and whatever mutation will follow.
LikeLike
This is an interesting story. We need to get together a compendium of stories people have personally experienced that shine light on the way testing creates an industry. Yours is fascinating.
LikeLike
Pearson, Roy, was in BIG trouble. Textbook sales were way, way down. But then they investing heavily in testing–buying testing companies, and now they are exporting this standards-and-testing model worldwide. BIG return on investment in this because the black-and-white paper tests are really cheap to produce (very low paper, printing, and binding costs) and the online ones cost almost nothing to scale.
LikeLike
My district now mandates we use Khan as part of SAT prep for juniors and it is written into our goals. We’ll see if practice SAT scores (given quarterly in addition to taking the PSAT and multiple short practice sessions) go up with use of Khan. The director says it isn’t mandatory but made us use it in goals. I’d say that’s mandatory. So much time spent on this crap!
LikeLike
What type of student will use the Khan Academy to prep for the SAT?
I’ll tell you. A motivated student who probably comes from a home and family where reading and education is important and poverty is not an issue.
If a child or teen didn’t grow up in a home where anyone reads and there are no books, newspapers and magazines and that family lives in poverty, the odds are against that child being motivated enough to use the Khan Academy. In fact, most of the children that life in that world if they take the SAT will take it without any prep gambling that their wild guessing will end up giving them a high score even if they can’t read.
LikeLike
Once upon a time, 25 years ago, I ‘offered’ SAT tutoring (at a rather high price of $50/hr.) to denizens of a tony private school. I could charge that much because I ‘got results’. But, it was rather easy to improve scores.
First, there was the fact that almost all of my clients had scored ‘too low’ when they first took the test. The probability was, therefore, that if they simply took it again, they would ‘improve’. Secondly, most low performers had a certain level of anxiety when they took the test. Simply being familiar with the format by reviewing former tests helped those students assess the test in a more calm and analytical manner. Thirdly, despite the subtraction of ‘wrong’ answers from the score (at a rate commensurate with the number of answers), the students needed to understand that they actually knew something, if only at the subconscious level, and they needed to ‘guess’ (even randomly) because an inaccurate random guess didn’t really count against the score. They needed to trust their instincts.
The result was often (among ‘median’ scores) a 100 point increase. Were the students any ‘smarter’ after the tutoring? Well, no. Were they more ‘scholastically fit’, well, no. All they learned was how to contain their anxiety and (to some extent) ‘psyche out’ the test. Nothing more.
The test is (and always was) a scam. I say this as one who has benefited from such tests as a youth (I always did well, even qualified for MENSA on my GRE’s). These tests measure nothing of value, and I’m ever so happy to see more and more colleges relying on a body of student work and the recommendations of former teachers (sometimes in the form of ‘grades’… although ‘grades’ are only a shapshot in time, and often a narrowly forced evaluation by a particular teacher who would have much more to say, if asked).
LikeLike
“even qualified for MENSA on my GRE’s”
Would you please explain that Daedalus? I didn’t know GRE scores could get one qualified for MENSA. What score is/was needed to get that designation? Was the GRE used in conjunction with other scores? etc. . . .
LikeLike
All 3 of my kids, tho unusual (& different from each other in learning style), had very mediocre SAT scores [10-15 yrs ago]. Only one chose to do private prep & re-take the test. [A waste: his score went up a little in math & down a little in verbal]. All were techie musicians & attended small specialized college programs where auditions counted most & grades a close 2nd. I suspect SAT scores were barely looked at: what those admissions folks needed to know was whether a musician, however talented, could make it through core courses.
Friends have a son who was a preternatural fact-collector/ retainer, loved competition/ being quizzed, aced tests. When he was 7 I was sure he’d be a lawyer. Sadly, bipolar I emerged; while learning to manage the illness he had a heck of a time with college and getting established in the working world. He finally got a toe-hold as a part-time SAT coach, turned out to good at it, succeeded w/his own test-prep biz. I am happy test-mania supports him, but he proves the point that a high SAT-score tells little beyond facility w/testing
LikeLike
Someone should give all of those — including administrators and teachers — who promote the ACT and the SAT as good measures of “college readiness” and “intelligence” a good swift kick in the seat of their pants. We are told by the test manufacturers that they are very good predictors of college success. Indeed colleges say that’s why they use them. But it simply isn’t true.
The authors of a study in Ohio found the ACT has minimal predictive power. For example, the ACT composite score predicts about 5 percent of the variance in freshman-year Grade Point Average at Akron University, 10 percent at Bowling Green, 13 percent at Cincinnati, 8 percent at Kent State, 12 percent at Miami of Ohio, 9 percent at Ohio University, 15 percent at Ohio State, 13 percent at Toledo, and 17 percent for all others. Hardly anything to get all excited about.
Here is what the authors said about the ACT in their concluding remarks:
“…why, in the competitive college admissions market, admission officers have not already discovered the shortcomings of the ACT composite score and reduced the weight they put on the Reading and Science components. The answer is not clear. Personal conversations suggest that most admission officers are simply unaware of the difference in predictive validity across the tests. They have trusted ACT Inc. to design a valid exam and never took the time (or had the resources) to analyze the predictive power of its various components. An alternative explanation is that schools have a strong incentive – perhaps due to highly publicized external rankings such as those compiled by U.S. News & World Report, which incorporate students’ entrance exam scores – to admit students with a high ACT composite score, even if this score turns out to be unhelpful.”
The SAT is no better. College enrollment specialists say that their research finds the SAT predicts between 3 and 14 percent of the variance in freshman-year college grades, and after that that nothing. Shoe size would work as well, or better.
The SAT is used by colleges for their own nefarious reasons, like boosting their rankings in “best” colleges lists and leveraging financial aid (increasingly and especially for students who don’t need it).
Here’s Princeton Review founder John Katzman on the SAT (and Princeton Review does quite a bit if test prep for the SAT):
“The SAT is a scam. It has been around for 50 years. It has never measured anything. And it continues to measure nothing. And the whole game is that everybody who does well on it, is so delighted by their good fortune that they don’t want to attack it. And they are the people in charge. Because of course, the way you get to be in charge is by having high test scores. So it’s this terrific kind of rolling scam that every so often, somebody sort of looks and says–well, you know, does it measure intelligence? No. Does it predict college grades? No. Does it tell you how much you learned in high school? No. Does it predict life happiness or life success in any measure? No. It’s measuring nothing. It is a test of very basic math and very basic reading skill. Nothing that a high school kid should be taking.”
http://www.pbs.org/wgbh/pages/frontline/shows/sats/interviews/katzman.html
Here’s The Big Test author Nicholas Lemann on the SAT:
“The test has been, you know, fetishized. This whole culture and frenzy and mythology has been built around SATs. Tests, in general, SATs, in particular, and everybody seems to believe that it’s a measure of how smart you are or your innate worth or something. I mean, the level of obsession over these tests is way out of proportion to what they actually measure. And ETS, the maker of test, they don’t actively encourage the obsession, but they don’t actively discourage it either. Because they do sort of profit from it…every time somebody takes an SAT, it’s money to the ETS and the College Board. But there is something definitely weird about the psychological importance these tests have in America versus what they actually measure. And indeed, what difference do they make? Because, there’s two thousand colleges in the United States, and 1,950 of them are pretty much unselective. So, the SAT is a ticket to a few places.”
http://www.pbs.org/wgbh/pages/frontline/shows/sats/interviews/lemann.html
The ACT and the SAT are used for the purpose of “financial-aid leveraging.” Instead of using a $20,000 scholarship for one needy student, schools can break that amount into four $5,000 grants for wealthier students who score higher, who will pay the rest of the tuition ($15,000 a year) and who will bring the school more cash and “will improve the school’s profile and thus its desirability.”
As Matthew Quirk wrote, “The ACT and the College Board don’t just sell hundreds of thousands of student profiles to schools; they also offer software and consulting services that can be used to set crude wealth and test-score cutoffs, to target or eliminate students before they apply…That students are rejected on the basis of income is one of the most closely held secrets in admissions; enrollment managers say the practice is far more prevalent than most schools let on.”
See: http://www.theatlantic.com/magazine/archive/2005/11/the-best-class-money-can-buy/4307/
None of this is really new.
For example, the 1992 report ‘Testing in American Schools.’ pointed out that “tests have too often been used to serve functions for which they were not designed or adequately validated.” Even Henry Goddard, who pushed intelligence testing relentlessly, later admitted that testing “could be perceived as justifying the richness of the rich and the poverty of the poor; they legitimized the existing social order.”
And, as Richard Rothstein reported nearly a decade-and-a-half ago in the American School Board Journal:
“Twenty years ago, Betty Hart and Todd Risley, two researchers from the University of Kansas, visited families from different social classes to monitor the conversations between parents and toddlers. Hart and Risley found that, on average, professional parents spoke more than 2,000 words per hour to their children, working-class parents spoke about 1,300, and welfare mothers spoke about 600. So by age 3, the children of professionals had vocabularies that were nearly 50 percent greater than those of working class children and twice as large as those of welfare children…Deficits like these cannot be made up by schools alone, no matter how high the teachers’ expectations. For all children to achieve the same goals, the less advantaged would have to enter school with verbal fluency that is similar to the fluency of middle-class children.”
Indeed, test scores are merely proxies for family income.
We’ve known for some time that poverty affects the developing brains of children. New research confirms it. A recent study at MIT “found differences in the brain’s cortical thickness between low-income and higher-income teenagers.” Not surprisingly, those differences found their way into test scores. Another recent study found that low-income children had brain surface areas 6 percent smaller than those of upper-middle class kids. That typically translates into less brain density, and then into less ability for ” language, memory, spatial skills and reasoning.”
Again, this is not new “news.” But it is what we ought to be focusing on. Common Core standards and testing will not alleviate it. Nor will ACT or SAT tests.
We have an awful lot to learn about improving public education. Brain development ought to be at the very center of the discussion. and as recent political developments prove, so should educating for democratic citizenship.
LikeLike
Here’s David Berliner on the problem of associating test scores with “effectiveness”:
“The roots of America’s educational problems are in the numbers of Americans who live in poverty. America’s educational problems are predominantly in the numbers of kids and their families who are homeless; whose families have no access to Medicaid or other medical services. These are often families to whom low-birth-weight babies are frequently born, leading to many more children needing special education… educational problems have their roots in families where food insecurity or hunger is a regular occurrence, or where those with increased lead levels in their bloodstream get no treatments before arriving at a schools’ doorsteps.”
https://www.washingtonpost.com/education/2018/10/22/education-professor-my-students-asked-who-i-would-vote-heres-what-i-told-them/?utm_term=.33ae8a253006
LikeLike
Very informative roundup, thanks!
Just one niggle: it’s intuitive that brain scans would reflect learning devpt, but what does that add to the discussion? Changes in brain surface area & cortical thickness do, in fact, seem to track intelligence over time, but science is far from a definitive grasp of the patterns, let alone the processes involved. https://academic.oup.com/cercor/article/25/6/1608/301081
A study showing similar results when tracking income level is suggestive, but has less probative value than the Hart/ Risley study you describe. Putting “brain development at the very center of the discussion” seems premature, & could encourage a new wave of assessment-happy profiteers looking to add brain scans to the toolkit.
LikeLike
bethree5,
there will always be hucksters….many of them have promoted the right brain-left brain nonsense that has suckered many educators.
another way to put it is in what brain researchers- like Marian Diamond – discovered with rat brains….rats that were raised in enriched environments developed brains that were bigger and denser than rats raised in a control environment, which had bigger, denser brains than those raised in an impoverished environment.
people are no different, and that’s the point that David Berliner makes.
LikeLike
What no Wilson???
THE most definitive takedown of the standards and testing malpractice regime??????
For those that aren’t aware of Noel Wilson’s seminal treatise, THE MOST important piece of educational writing in the last half-century, I direct readers to:
“Educational Standards and the Problem of Error” found at: http://epaa.asu.edu/ojs/article/view/577/700
Brief outline of Wilson’s “Educational Standards and the Problem of Error” and some comments of mine. (updated 6/24/13 per Wilson email)
A description of a quality can only be partially quantified. Quantity is almost always a very small aspect of quality. It is illogical to judge/assess a whole category only by a part of the whole. The assessment is, by definition, lacking in the sense that “assessments are always of multidimensional qualities. To quantify them as unidimensional quantities (numbers or grades) is to perpetuate a fundamental logical error” (per Wilson). The teaching and learning process falls in the logical realm of aesthetics/qualities of human interactions. In attempting to quantify educational standards and standardized testing the descriptive information about said interactions is inadequate, insufficient and inferior to the point of invalidity and unacceptability.
A major epistemological mistake is that we attach, with great importance, the “score” of the student, not only onto the student but also, by extension, the teacher, school and district. Any description of a testing event is only a description of an interaction, that of the student and the testing device at a given time and place. The only correct logical thing that we can attempt to do is to describe that interaction (how accurately or not is a whole other story). That description cannot, by logical thought, be “assigned/attached” to the student as it cannot be a description of the student but the interaction. And this error is probably one of the most egregious “errors” that occur with standardized testing (and even the “grading” of students by a teacher).
Wilson identifies four “frames of reference” each with distinct assumptions (epistemological basis) about the assessment process from which the “assessor” views the interactions of the teaching and learning process: the Judge (think college professor who “knows” the students capabilities and grades them accordingly), the General Frame-think standardized testing that claims to have a “scientific” basis, the Specific Frame-think of learning by objective like computer based learning, getting a correct answer before moving on to the next screen, and the Responsive Frame-think of an apprenticeship in a trade or a medical residency program where the learner interacts with the “teacher” with constant feedback. Each category has its own sources of error and more error in the process is caused when the assessor confuses and conflates the categories.
Wilson elucidates the notion of “error”: “Error is predicated on a notion of perfection; to allocate error is to imply what is without error; to know error it is necessary to determine what is true. And what is true is determined by what we define as true, theoretically by the assumptions of our epistemology, practically by the events and non-events, the discourses and silences, the world of surfaces and their interactions and interpretations; in short, the practices that permeate the field. . . Error is the uncertainty dimension of the statement; error is the band within which chaos reigns, in which anything can happen. Error comprises all of those eventful circumstances which make the assessment statement less than perfectly precise, the measure less than perfectly accurate, the rank order less than perfectly stable, the standard and its measurement less than absolute, and the communication of its truth less than impeccable.”
In other words all the logical errors involved in the process render any conclusions invalid.
The test makers/psychometricians, through all sorts of mathematical machinations attempt to “prove” that these tests (based on standards) are valid-errorless or supposedly at least with minimal error [they aren’t]. Wilson turns the concept of validity on its head and focuses on just how invalid the machinations and the test and results are. He is an advocate for the test taker not the test maker. In doing so he identifies thirteen sources of “error”, any one of which renders the test making/giving/disseminating of results invalid. And a basic logical premise is that once something is shown to be invalid it is just that, invalid, and no amount of “fudging” by the psychometricians/test makers can alleviate that invalidity.
Having shown the invalidity, and therefore the unreliability, of the whole process Wilson concludes, rightly so, that any result/information gleaned from the process is “vain and illusory”. In other words start with an invalidity, end with an invalidity (except by sheer chance every once in a while, like a blind and anosmic squirrel who finds the occasional acorn, a result may be “true”) or to put in more mundane terms crap in-crap out.
And so what does this all mean? I’ll let Wilson have the second to last word: “So what does a test measure in our world? It measures what the person with the power to pay for the test says it measures. And the person who sets the test will name the test what the person who pays for the test wants the test to be named.”
In other words it attempts to measure “’something’ and we can specify some of the ‘errors’ in that ‘something’ but still don’t know [precisely] what the ‘something’ is.” The whole process harms many students as the social rewards for some are not available to others who “don’t make the grade (sic)” Should American public education have the function of sorting and separating students so that some may receive greater benefits than others, especially considering that the sorting and separating devices, educational standards and standardized testing, are so flawed not only in concept but in execution?
My answer is NO!!!!!
One final note with Wilson channeling Foucault and his concept of subjectivization:
“So the mark [grade/test score] becomes part of the story about yourself and with sufficient repetitions becomes true: true because those who know, those in authority, say it is true; true because the society in which you live legitimates this authority; true because your cultural habitus makes it difficult for you to perceive, conceive and integrate those aspects of your experience that contradict the story; true because in acting out your story, which now includes the mark and its meaning, the social truth that created it is confirmed; true because if your mark is high you are consistently rewarded, so that your voice becomes a voice of authority in the power-knowledge discourses that reproduce the structure that helped to produce you; true because if your mark is low your voice becomes muted and confirms your lower position in the social hierarchy; true finally because that success or failure confirms that mark that implicitly predicted the now self-evident consequences. And so the circle is complete.”
In other words students “internalize” what those “marks” (grades/test scores) mean, and since the vast majority of the students have not developed the mental skills to counteract what the “authorities” say, they accept as “natural and normal” that “story/description” of them. Although paradoxical in a sense, the “I’m an “A” student” is almost as harmful as “I’m an ‘F’ student” in hindering students becoming independent, critical and free thinkers. And having independent, critical and free thinkers is a threat to the current socio-economic structure of society.
LikeLike
Thus,
“The reason-defying power of ideological (TRIBAL) thinking, is such that
people are able to deny not only elementary morality-DO NO HARM
(that you should prevent the worst possible outcome)
but even elementary arithmetic. It’s remarkable.” Chris Wright
Testing is HARM.
…Just following (testing)orders dictated by the unelected.
“The evaluation rubric presses teachers to teach to the test every day.”
…Public Schools are NOT failing.
“…millions of kids are being made to plow through even more test prep, and the opportunity costs of this are enormous.”
elementary arithmetric: testing costs for 85% (Public School ) vs
costs of the 15% “choice” schools…
Damn “choice” schools…
It’s remarkable…
LikeLike
Not only remarkable but sad, disgusting, unethical and an abomination of educational malpractice.
Far too few understand what you are saying NoBrick.
LikeLike
Something important seems to have been lost from the “get-go” in the Comments string above (although I stopped reading after plowing through several pages of the same “Khan Academy is selling its soul” remarks).
The SAT test prep service is being offered for FREE.
I dislike standardized testing as much as most readers of this blog, but please don’t forget that parents are being gouged by commercial test prep companies and private tutors every day of the year!
Like it or not, students for the near future at least will continue to take this test and at least now there is an option that ALL families can afford.
Full Disclosure: I am retired and tutor students in math, physics and chemistry, but I refuse to do test prep work.
LikeLike
David, do you use the Khan Academy materials for test prep?
LikeLike
Hi Diane.
As I said at the end of my reply, I do NOT do test prep in my tutoring practice.
From everything I see in my tutoring work, the constant testing and AP classes tend to kill the love of science and math, not promote them, and I have been writing about this problem on my blog at eduissues.com for the last two plus years.
Regarding Khan Academy itself, although Mr. Khan is an excellent lecturer, that is all that I have usually seen there – one way instruction with limited practice problems. For kids who don’t respond well to that (probably the majority), I think that the site is not that much more helpful than having another textbook. Unfortunately the art of self-learning from textbooks also seems to be on the way out…
Despite the defects of the site though, at least it offers less wealthy families an option that they didn’t have before.
LikeLike