In the past few days, the media has barraged us with stories about how American students rank on PISA’s “problem-solving” test. We were told that they scored better than average yet still behind other nations.
But what is the test and what does it mean?
Andy Hargreaves of Boston College, co-author with Michael Fullan of Professional Capital, tweeted to me an article in the British press that contains examples taken from the test.
As I read the questions, I am reminded of standardized test questions I have seen that pop up on tests of reason and logic or on IQ tests.
Why don’t we administer the PISA problem-solving test to our state legislators and publish their scores? Or to the top officials at the U.S. Department of Education?
Now that would be interesting, wouldn’t it?
Diane Ravitch's blog 
Why don’t we administer the PISA problem-solving test to our state legislators and publish their scores? Or to the top officials at the U.S. Department of Education?
Oh yes yes yes. That would be a hoot.
A fair question: if those tests actually measure something important, then the first people who should take them in the USofA—
Arne Duncan, John Deasy, John King, Michelle Rhee, David Coleman, Paul Vallas, Bill Gates, Jeb Bush, Candace McQueen, Wendy Kopp and the rest of that merry band of $tudent $ucce$$ers.
Of course, under controlled secure conditions. Wouldn’t want any of them succumbing to the urge to cheat, now would we? That would Rheeally suck…
😎
Erase to the top!
Dear David and Arne. What do you think of this idea? Let’s say you and I and KrazyTA here all take these tests. We could broadcast the testing in real time in the same time slot as, say, American Idol. No more Waiting for Superman. You could show up in your Superman suits. Spiff.
If Krazy and I get the higher scores, you scrap the embarrasingly amateurish Common Core and the abusive, numerological, junk science tests and VAM.
If you get the higher scores, you continue with your sick policies until the people of the country get wise to you and drive you off to hide out for the rest of your careers in some Plutocrat-subsidized “think” tank.
Deal?
Good idea.
But what about having them take the test online using an iPad. That way every time they answered a question, the results would be displayed for the whole world to watch in real time.
And once the majority of people discover what they are up to, they won’t go into hiding in the US. They’ll ask for asylum from Putin in Russia.
Not only that, Diane, but the tests themselves do not take into consideration the huge discrepancies in educational systems, the tracking that some nations do to limit those that stay on an academic track while we here in the US attempt to educate everyone. In Germany for example, some students END their public school education in 9th grade and never make it into the pool for PISA.
Going further, in past years less than 2,000 US students took the PISA exams (in 2012 they raised it to just over 6K) and these are supposed to be a representative sample of US students? The 2012 US “representative sample” came from only three states – MA, CT and FL.
Do we even want to talk about how Shanghai limits their education system to exclude migrant families?
Far too many countries play fast and dirty with their representative samples not to mention the questionable scoring – giving students a reading score when only 50% actually had reading questions?
The fix is certainly on and Bill Gates and his ilk are using this to steamroll public money into the private sector!
UnCommonMaryAnn: you point to a crucial factor.
“Principle of Data Interpretation: When comparing groups, make sure the groups are comparable.”
[Gerald Bracey, READING EDUCATIONAL RESEARCH: HOW TO AVOID GETTING STATISTICALLY SNOOKERED, 2006, p. 31]
Unfortunately, when the charterites/privatizers seize control of a conversation, thoughtfulness and critical thinking get tossed out the window.
Thank you for your comments.
😎
If we gave the PISA test to Congress and they did poorly, they’d blame it on the public schools they attended and demand we fire all the teachers.
I love the idea of administering the tests to legislators. You brought a smile to my face. Thanks for that.
First of all, are the scores significantly different enough to rank? Second, is it possible that American student’s problem solving is slipping because of NCLB and RttT and that other countries are gaining because they are becoming more like we used to be?
I think these kinds of problems actually do measure something important. Whether and how that “something” can be taught and learned (apart from the teaching of test-taking strategies), I don’t know.
What do you get if you take the E out of OECD?
Suppose that you are a Plutocrat Investor attending a conference on THE NEXT BIG INVESTMENT OPPORTUNITY.
One Edupreneur has a virtual online charter school company; this company is poised to make Six Gazillion dollars over the next year by bilking taxpayers and taking advantage of sweetheart crony deals with political buddies.
Another Edupreneur has an educational software company that specializes in Big Data; this company is poised to make only Two Gazillion dollars over the next year by bilking taxpayers and taking advantage of sweetheart crony deals with political buddies.
Another Edupreneur is running a startup VAM services provider; this company is poised to make only One Gazillion dollars over the next year by bilking taxpayers and taking advantage of sweetheart crony deals with political buddies.
The three Edupreneurs are named Tina, Rita, and Carlita.
You don’t know which is which, but to make the investment interesting, the conference organizer has required you to make a blind choice about which to invest in.
You choose Tina.
The conference organizer then tells you that Rita has the online charter school company, and you can no longer invest there.
The organizer then says to you, “Do you want to choose Carlita?”
Should you stick with Tina or switch to Carlita?
Which will maximize your potential to bilk taxpayers and take advantage of sweetheart crony deals with political buddies?
Questions seem ridiculous to me. Let’s solve this problem instead : politicians and profit making ventures take over and start to ruin education. How do we stop them?
Better question!!!!
Christine gets an A.
🙂
Answer: open source alternatives
These questions are very similar to those on the analytical section of the old GREs (on which I scored 800) and IQ tests (I won’t divulge that score here). It seems to me they measure raw brain power. I don’t think this is something we can teach. Am I wrong? Can anyone explain to me how we can teach analytical ability or teach IQ?
The SBAC/PARCC tests seem to be modeled on the PISA. They demand this kind of laborious mental heavy lifting. What’s being employed is not knowledge gained through schooling, but rather hard-wiring of the brain and one’s doggedness. Such tests are spawning curricula that resemble on-line “brain training” programs like Lumosity rather than rich liberal arts curricula. These ill-conceived tests are going to drive a stake in the heart of the liberal arts (already floundering under a century of attacks).
These are graph theory problems. Procedures for solving them can definitely be taught.
Thanks for the response, Robert. I guess I should stop teaching history and start teaching graph theory problems.
http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg
LOL. The interesting thing is that these people who concentrate on instruction in skills as opposed to content, who believe that that’s what we should be teaching, usually, themselves, know very little about the skills areas that they are addressing. This is a case in point. The folks who put these problems together probably thought that they were general problem-solving problems and did not recognize that there are actually well established procedures for solution of such problems. This is EMPHATICALLY true of most of the crap that goes under the heading of “Critical Thinking” instruction. Almost all of what is done there is ignorant nonsense. There are many arts and sciences devoted to clear thinking (propositional logic, predicate logic, modal logic, set theory, probability, statistics, philosophy of science) taht have real techniques that are applicable to “critical thinking,” and if one were going to teach that per se, then one would draw upon those, but most of the people writing and teaching these critical thinking lessons know nothing of them, and so the lessons they create are worthless. The child usually finishes the critical thinking lesson having learned not one concrete, useful technique for thinking “critically.”
This nonsense of instruction in skills per se divorced from content, the reductio ad absurdum of which we are seeing in the ridiculous PARCC and SBAC ELA tests and in these most recent NY ELA tests, is a disease that has absolutely permeated our system. Instead of teaching literature and history and art and science and philosophy, we teach skills for use by students of literature, history, etc. People forget that one reads a novel because it is of value in itself. The novel becomes an interchangeable opportunity for exercise of novel-reading skills, and the teaching concentrates on those. Compare the execrable crap about, say, the Puritan Era that one finds in a contemporary basal American literature book to the material on that era in a textbook from thirty years ago. Thirty years ago, the student would come away from the unit understanding lots of ideas, about Calvinism and election and predestination and Original Sin and local governance and the primacy of the Word and the Protestant Ethic and salvation through grace as opposed to salvation via intermediaries, sacraments, and indulgences. And all of that would help the student to understand that current in American thought that has run down through our history to the fundamentalist churches today–to the Rove phenomenon in U.S. politics, the religious wing of the Tea Party, etc., because to understand our past is to understand our present. But the student today gets none, or almost none, of that in the current best-selling American lit basals, which are all skills all the time. The intellectual meat has been gutted and replaced with vague activities for developing skills that, of course, do no such thing because skill is dependent upon knowledge.
Well said! Everyone’s using the buzz word “critical thinking” and almost no one has a clear idea what he’s talking about.
Such tests are spawning curricula that resemble on-line “brain training” programs like Lumosity rather than rich liberal arts curricula. These ill-conceived tests are going to drive a stake in the heart of the liberal arts
yes yes yes
The PISA examples are math and logic problems. They are not tests of creativity. Look up the tests and informed theoretical work of Joseph W. Getzels and E. Paul Torrance.
The Torrance tests, available from Scholastic http://ststesting.com/2005giftttct.html, are most often used to identify children, adults, and “special populations” as gifted. The pictorial and verbal tests measure three strengths in thinking: fluency, flexibility, and originality. In the figural tests, participants create simple drawings and respond to images. Scores are derived from evidence of qualities such as elaboration, expressiveness, storytelling, humor, and fantasy.
Relatively few people are aware that the Council of Chief State School Officers (CCSSO) is a developing a web-based scale for measuring creativity, one of several in the “EdSteps” project—funded by the Bill and Melinda Gates Foundation and operated by the CCSSO
EdSteps had a low profile until July, 2010 when Newsweek announced a “The Creativity Crisis,” citing a steady decline in scores on the Torrance Tests of Creativity since 1990. The tests are widely used and respected, in part, because records have been kept on the childhood scores and the later-in-life creative accomplishments of each cohort of test takers since the late 1950s (e.g., citations in art publications, patents and awards, books and articles published).
In response to inquiries, the CCSSO issued a press release that dismissed the Torrance tests and referred its own work on creativity, emphasizing that EdSteps is a project to “advance creativity to the highest possible international standards, and measure creativity in a way that is situated in a context of actual activity.” Creativity is defined as “the valued uses and outcomes of originality driven by imagination, invention, and curiosity.”
The Edsteps creativity scale a work-in process. The website solicits work samples on any subject from people of all ages and abilities, “globally”…”in any form, genre, or media”…” “writing, videos, images, charts, or other graphics.” People who visit the site are asked to compare two submissions and decide which is the most “effective” (undefined, but the favorite word of Bill Gates).
That process is carried out in multiple iterations, by multiple judges, with multiple examples. This process is supposed to result in a scale representing a progression of achievement from novice to expert, without the need for written criteria or explanations.
The process is not different from a popularity contest, with samples of work identified by age, gender, ability level, geographic region, type of work, and the like.
I could not discover how the EdSteps addresses this fact: Works created by children can be judged more creative than work produced by well-trained adults (e.g., a quote attributed to Picasso: ”It took me four years to paint like Raphael, but a lifetime to paint like a child”). I cannot imagine how a single scale of creative achievement can be constructed from an “anything goes” basket of work from around the world, subject to further editing by EdSteps into web-friendly snippets. The release forms for the project are horrific.
I think this effort is a crock, but I could be wrong. My sources: Bronson, P. & Merryman, A. (2010, July 10). The creativity crisis. Newsweek. Retrieved from http://www.newsweek.com/2010/07/10/the-creativity-crisis.html .///. Council of Chief State School Officers (2010). CCSSO response to ‘The creativity crisis.’ Press release Retrieved from http://www.ccsso.org/News_and_Events/Current_News/CCSSO_Response_to_the_Creativity_Crisis.html //// EdSteps. (2010b). February 22). Developing the EdSteps continuum: Report of the EdSteps technical advisory group. Retrieved from http://www.edsteps.org/CCSSO/DownloadPopUp.aspx?url=SampleWorks/EdStepsScalingApproach_Long.pdf // EdSteps. (2011b). Creativity launches. Retrieved from http://www.edsteps.org/CCSSO/ManageContent.aspx?system_name=nP6iGdNaft7MEwLG6uDXXA==&selected_system_name=DRkDdjiObdU=
I posted the other night on some resources designed to teach the joy of learning. This creativity testing strikes me as something along the same line. Policy gurus have spent the last couple of decades killing creativity and the joy of learning. Now they are going to figure out some surrogate measure for joy and creativity for which they will then test! Is something wrong with this picture?
What is wrong indeed.
These are definitely NOT tests of creative thinking generally or of divergent thinking in particular.
Creativity through regimentation and standardization!!! Creative following of the creativity rubric! LOL