Archives for category: NAEP

The new Common Core tests funded by the federal government agreed to adopt the standard of “proficiency” used by the National Assessment of Educational Progress (NAEP). Students who are not “proficient” are deemed to have “failed” to meet the standards. They are described as “not proficient,” which is a very bad thing indeed.

But what does NAEP proficiency mean?

I served on the NAEP governing board for seven years. I understood that “proficient” was a very high standard. There are four NAEP achievement levels: Advanced (typically reached by 5-8% of students); Proficient (typically reached by about 35-40% of students); Basic (typically reached by about 75% of students); and Below Basic (very poor performance, about 20-25% of students). Thus, by aligning its “pass” mark with NAEP proficient, the PARCC and SBAC (the two testing groups) were choosing a level that most students will not reach. Only in Massachusetts have as many as 50% of students reached NAEP proficient. Nearly half have not.

As Catherine Gewertz wrote in Education Week, “The two common-assessment consortia are taking early steps to align the “college readiness” achievement levels on their tests with the rigorous proficiency standard of the National Assessment of Educational Progress, a move that is expected to set many states up for a steep drop in scores.
After all, fewer than four in 10 children reached the “proficient” level on the 2013 NAEP in reading and math.”

So, if these consortia intend to align with the very rigorous standards of NAEP, most students will fail the tests. They will fail them every year. Will the test results be used for promotion and/or graduation? If so, we can expect a majority of the current generation of students not to be promoted or graduate from high school. What will we do with them?

It is time to ask whether NAEP proficient is the right “cut score” (passing mark). I think it is not. To me, given my knowledge of NAEP achievement levels, proficient represents solid academic performance, a high level of achievement. I think of it as an A. Advanced, to me, is A+. Anyone who expects the majority of students to score an A on their state exams is being, I think, wildly unrealistic. Please remember that NAEP proficient represents a high level of achievement, not a grade level mark or a pass-fail mark. NAEP basic would be a proper benchmark as a passing grade, not NAEP proficient.

Furthermore, the NAEP achievements levels have been controversial ever since they were first promulgated in the early 1990s when Checker Finn was chairman of the NAEP governing board. Checker was subsequently president of the Thomas B. Fordham Foundation/Institute, and he has long believed that American students are slackers and need rigorous standards (as a member of his board for many years, I agreed with him then, not now). He believed that the NAEP scale scores (0-500) did not show the public how American students were doing, and he was a strong proponent of the achievement levels, which were set very high.

James Harvey, a former superintendent who runs the National Superintendents’ Roundtable, wrote an article in 2011 that explains just how controversial the NAEP achievement levels are.

He wrote then:

Since definition is crucial in any discussion of standards, let’s define the terms of the discussion. The No Child Left Behind Act, passed by Congress in 2001 as the latest reauthorization of the Elementary and Secondary Education Act of 1965, permitted states to develop their own assessments and set their own proficiency standards to measure student achievement. Most states, for their purposes, quite sensibly defined proficiency as performance at grade level.

What about NAEP? Oddly, NAEP’s proficient standard has little to do with grade-level performance or even proficiency as most people understand the term. NAEP officials like to think of the assessment standard as “aspirational.” In 2001, long before the current contretemps around state assessments, two experts associated with the National Assessment Governing Board—Mary Lynne Bourque, staff member to the governing board, and Susan Loomis, a member of the board—made it clear that “the proficient achievement level does not refer to ‘at grade’ performance. Nor is performance at the proficient level synonymous with ‘proficiency’ in the subject. That is, students who may be considered proficient in a subject, given the common usage of the term, might not satisfy the requirements for performance at the NAEP achievement level.”

It is hardly surprising, then, that most state assessments aimed at establishing proficiency as “at grade” produce results different from a NAEP standard in which proficiency does not refer to “at grade” performance or even describe students that most would think of as proficient. Far from supporting the NAEP proficient level as an appropriate benchmark for state assessments, many analysts endorse the NAEP basic level as the more appropriate standard because NAEP’s current standard sets an unreasonably high bar.

What is striking in reviewing the history of NAEP is how easily its governing board has shrugged off criticisms about the board’s standards-setting processes.

In 1993, the National Academy of Education argued that NAEP’s achievement-setting processes were “fundamentally flawed” and “indefensible.” That same year, the General Accounting Office concluded that “the standard-setting approach was procedurally flawed, and that the interpretations of the resulting NAEP scores were of doubtful validity.” The National Assessment Governing Board, or NAGB, which oversees NAEP, was so incensed by an unfavorable report it received from Western Michigan University in 1991 that it looked into firing the contractor before hiring other experts to take issue with the university researchers’ conclusions that counseled against releasing NAEP scores without warning about NAEP’s “conceptual and technical shortcomings.”

“Most state assessments aimed at establishing proficiency as ‘at grade’ produce results different from a NAEP standard.”
In addition, NAGB absorbed savage criticism from the National Academy of Sciences, which concluded in 1999 that “NAEP’s current achievement-level-setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters’ judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results. … The results are not believable.”

For the most part, such pointed criticism has rolled off the governing board like so much water off a duck’s back.
As recently as 2009, the U.S. Department of Education received a report on NAEP from the University of Nebraska’s Buros Institute. This latest document expressed worries about NAEP’s “validity framework” and asked for a “transparent, organized validity framework, beginning with a clear definition of the intended and unintended uses of the NAEP assessment scores. We recommend that NAGB continue to explore achievement-level methodologies.” In short, for the last 20 years, it has been hard to find any expert not on the Education Department’s payroll who will accept the NAEP benchmarks uncritically.

Those benchmarks might be more convincing if most students outside the United States could meet them. That’s a hard case to make, judging by a 2007 analysis from Gary Phillips, a former acting commissioner of the National Center for Education Statistics. Phillips set out to map NAEP benchmarks onto international assessments in science and mathematics and found that only Taipei (or Taiwan) and Singapore have a significantly higher percentage of proficient students in 8th grade science than the United States does. In math, the average performance of 8th grade students in six jurisdictions could be classified as proficient: Singapore, South Korea, Taipei, Hong Kong, Japan, and Flemish Belgium. Judging by Phillips’ results, it seems that when average results, by jurisdiction, place typical students at the NAEP proficient level, the jurisdictions involved are typically wealthy—many with “tiger mothers” or histories of excluding low-income students or those with disabilities.

None of this is to say that the method of determining the NAEP achievement levels is entirely indefensible. Like other large-scale assessments—the International Mathematics and Science Study, the Progress on International Reading Literacy Survey, and the Program on International Student Assessment—NAEP is an extremely complex endeavor, depending on procedures in which experts make judgments about what students should know and construct assessment items to distinguish between student responses. Panels then make judgments about specific items, and trained scorers, in turn, bring judgment to bear on constructed-response items, which typically make up about 40 percent of the assessment.

That said, it is hard to avoid some obvious conclusions. First, NAEP’s achievement levels, far from being engraved on stone tablets, are administered, as Congress has insisted, on a “trial basis.” Second, NAEP achievement levels are based on judgment and educated guesses, not science. Third, the proficiency benchmark seems reachable by most students in only a handful of wealthy or Asian jurisdictions.

It is important to know this history when looking at the results of the Common Core tests (PARCC and SBAC). The fact that they have chosen NAEP proficient as their cut score guarantees that most students will “fail” and will continue to “fail.” Exactly what is the point? It is a good thing to have high standards, but they should be reasonable and attainable. NAEP proficient is not attainable by most students. Not because they are dumb, but because it is the wrong cut score for a state examination. It is “aspirational,” like running a four-minute mile. Some runners will be able to run a four-minute mile, but most cannot and never will. Virtually every major league pitcher aspires to pitch a no-hitter, but very few will do it. The rest will not, and they are not failures.

What parents and teachers need to know is that the testing consortia have chosen a passing mark that is inappropriate, that is not objective, and that is certain to fail most students. That’s not right, and that’s not fair.

New York State has bumbled into bizarre-O land. Chalkbeat reports that Néw York’s Common Core tests are more difficult than NAEP.

The NAEP tests are supposed to be internationally benchmarked. NAEP proficient is a very high standard that most students have never met (except in Massachusetts, where barely 50% reach proficient).

“In eighth-grade math, 22 percent of students earned what New York state called a passing score last school year, while 32 percent were deemed proficient on the NAEP exams. In fourth-grade reading, 33 percent passed the state test, while 37 percent of students earned a proficient score on the NAEP test. (Massachusetts was the other outlier, with more students earning a proficient score on the eighth-grade math NAEP test than on the state’s own tests.)”

State officials are pleased that their standards are beyond the reach of most students. For some strange reason, high failure rates are a source of pride. Bizarre.

The more they design tests to fail most students, the more the Opt Out movement will grow. When did education fall into the hands of technocratic sadists? They think education is a test of endurance, where only the stirring survive. Parents see education as a process of development, not a cruel race.

When I spoke to the Texas School Boards Association a few years ago, a member of the audience got up and identified himself as a school board member and an engineer. He said that he didn’t understand why the government tests every child every year. He said that in the industry where he works, it is customary to test the products periodically, on a sampling basis. I will never forget what he said: “If we tested every product, we would spend most of our time testing the product, and we wouldn’t have time left to manufacture or to improve the product.”

 

I was reminded of that statement when I received this comment from Doug Garnett, who is a specialist in marketing, advertising, branding, communications, and technology. Garnett wrote, just minutes ago:

 

Where I’m mystified is this belief that in order to have “accountability”, EVERY child has to be tested in the entire nation.

 

In business, we rely heavily on statistical sampling because it’s flat out too expensive to measure every item. Sampling in manufacturing, sampling in store satisfaction, sampling in purchasing, sampling in advertising impact, sampling, sampling sampling.

 

The NAEP relies on sampling…because it’s EFFECTIVE!

 

Imagine this: IF we shifted to a sampling test approach an amazing array of issues would be mitigated. The tests would lose their intensely punitive nature – and evolve toward being instructive and enlightening. They would lose the “high stakes” and become simply learning that informs. And, WE could use their reduced presence to focus on the totality of education instead of creating testing farms.

 

So…why don’t these so-called “business people” behind reform endorse smart business approach like sampling? Mind boggling…unless we embrace the conspiracy to redirect all of that government spending into the profits of private corporations.

According to Politico.com, Senator Lamar Alexander is considering eliminating the federal mandate for annual testing in grades 3-8. Charles Barone of the hedge fund managers’ “Democrats for Education Reform” is alarmed by this proposal, claiming it is an “equity” issue that would make it impossible to compare states.

Why the need to compare test scores is an equity issue is unexplained. Apparently Barone–who used to work for Congressman George Miller, senior Democrat on the House Education Committee–is unfamiliar with NAEP. That is the National Assessment of Educational Progress, which has been testing American students since 1969. It has been comparing states since 1992 and disaggregating scores by race, gender, language, and disability status. I hope proponents of annual testing will soon explain how comparing states creates equity. We know that Mississippi has lower scores than Massachusetts, whether we test annually or every three years. The gap is not changed by knowing about it more frequently but by funding schools attended by low-performing students so they can have smaller classes, more arts programs, more specialists, better paid professionals, and amply supplied and staffed libraries.

Here is the story:

THE GOP DRAFT YOU’VE BEEN WAITING FOR: Republican Sen. Lamar Alexander unveiled a discussion draft Tuesday night detailing his plan for reauthorizing No Child Left Behind. It borrows heavily from his 2013 proposal [http://1.usa.gov/1C4GZNS ] and if passed, it would take the federal government right out of some of the Obama administration’s most contentious policies – providing relief to states that haven’t met the administration’s bar for accountability systems and teacher evaluations. The bill would give states the option to make more than $14 billion in Title I funding portable across public schools. Alexander’s draft also makes clear that the federal government would have no involvement in states’ academic standards – although states would have to set high standards. When it comes to testing, one option would allow districts to forgo annual exams. Maggie Severns reports: http://politico.pro/14SYoPQ Read the discussion draft here: http://1.usa.gov/1swgqBH

– Some feel that testing option would make it impossible to compare results at the state level. “This, by extension, becomes an equity issue,” said Charles Barone, policy director for Democrats for Education Reform. “Any effort to advance equity requires comparability of student circumstances across zip codes, incomes, race, disability, etc. Any accountability system that drives to improve the achievement of those students and target resources toward them is out the window if every school or district is held accountable based on a different set of numbers.”

After four years of Governor Chris Christie, we are used to loud complaints about how terrible New Jersey’s schools are, how poorly they perform compared to Tennessee (Arne Duncan’s favorite), how expensive they are, how large the achievement gaps are.

Bruce Baker shows that none of this is true: New Jersey’s schools perform very well indeed, and they exceed expectations.

Baker documents what he says and concludes:

“To summarize:

“NJ schools do better than expected on NAEP given statewide poverty rates, ranking among the highest states.

“NJ schools have gained more on NAEP than nearly all other states (when correcting for starting point)

“NJ’s 8th grade achievement gaps are relatively average (when correcting for income gaps). The only NJ achievement gap that is greater than average is grade 8 reading.

“NJ’s 4th grade achievement gaps are among the smallest among states (when correcting for income gaps)

“So congratulations, NJ… you’re doin’ pretty well. That’s not to say by any stretch of the imagination that we should be complacent. We’ve still got Massachusetts to catch up to in most cases. They, not Tennessee or Louisiana are giving us a run for our money. And as I pointed out in my most recent post, we need to give serious consideration to reinvesting in our neediest communities. Prior investments (including early childhood programs) may provide partial explanation for why our fourth grade achievement gaps are so relatively small. But we’ve backed off substantially on funding fairness in recent years, the consequences of which are yet to be measured.”

Someone sent me this clip from Tennessee, where Arne Duncan was trying to salvage the federally-funded online Common Core test called PARCC.

“DUNCAN: TENNESSEE CAN STILL SALVAGE TESTS: At Brick Church College Prep in Nashville, Tenn., Education Secretary Arne Duncan showered the state with praise for becoming the fastest improving state in the country. But it still has a long way to go, he said after a town hall event [http://bit.ly/1tgEe8P ] with state chief Kevin Huffman. The legislature delayed Common Core-aligned PARCC tests for a year, but Tennessee has time for a fix, he said. “I think that having high standards is really important,” Duncan said. “Having an honest way to measure that you’re hitting those high standards and to have transparency across the country. So if all you’re able to do is measure Tennessee students against Tennessee students and not have any sense of how you’re doing versus Massachusetts or Kentucky or Mississippi, I think that misses the point. I think the state still has a chance to do the right thing going forward.”

Question: has Secretary of Education Duncan heard about the federally-funded National Assessment of Educational Progress? Since 1992, it has been measuring academic progress in the states. Using NAEP, it is possible to compare students in Tennessee to students in Massachusetts, Kentucky, Mississippi, and other states. Instead of testing every single student, it tests scientific samples in every state and nationally. It has no stakes attached. Isn’t that as much testing as we need to compare states?

A brilliant post by G.F. Brandenburg about NAEP scores.

Shows how little has been gained by the Bush-Obama demolition derby of testing, closing schools, firing teachers and principals, opening charters.

It is all a mighty failure that has not improved test scores or education

Hoax!

This NPR report summarizes the 12th grade NAEP report: Scores for high school seniors are flat. Reading scores in 2013 were lower than in 1992.

 

While there were small gains for each racial and ethnic group since 2005, there were no gains at all since 2009, when Race to the Top was initiated.

 

Achievement gaps among racial and ethnic groups remain wide.

 

Secretary of Education gnashed his teeth and said the results were troubling, and he is right. The chair of the National Assessment Governing Board said the results were unacceptable, and he is right.

 

In mathematics, the states that made the biggest gains in proficient students were: South Dakota, Massachusetts, New Jersey, Connecticut and New Hampshire. Only one of these–Massachusetts–won a Race to the Top award.

 

Also in mathematics, the states that had a lower percentage of proficient students than the rest of the nation were: Tennessee, Arkansas, West Virginia, and Florida. Two of the lowest performing states won Race to the Top awards: Tennessee and Florida.

 

In reading, the states that outperformed the nation were Idaho, Connecticut, Massachusetts, New Jersey, New Hampshire, Iowa, and South Dakota. Only one of these states–Massachusetts–won a Race to the Top award.

 

Also in reading, the states that had the lowest percentage of proficient students were: Tennessee, Arkansas, and West Virginia. Tennessee won a Race to the Top award.

 

These twelfth graders started school about the time that No Child Left Behind was signed into law, on january 8, 2002. Their entire school lives has been dominated by testing. The survival of their school depended on their test scores. Billions and billions of dollars have been diverted from classroom instruction to testing corporations. Many districts have increased class sizes and reduced services to students. Some leave closed libraries and laid off librarian, social workers, counselors, and psychologists. Many thousands of teachers have lost their job. But the testing industry has grown to be a multi-billion dollar enterprise, fattened by NCLB and RTTT.

 

Secretary Duncan is right. This is indeed troubling. It is time to change course. The policies of the Bush-Obama era have failed.

 

 

FairTest comments on today’s release of 12th grade NAEP scores. Their conclusion: a dozen years of test-based accountability has had no discernible effect on the test scores of seniors.

This accords with the 2010 report of the National Research Council, which released a report saying that incentives and test-based accountability are ineffective.

Although I have never put much stock in 12th grade NAEP results, due to lack of student motivation on a test that doesn’t count, there ought to be some residual effects of 12 years of testing, testing, testing.

 

 

FairTest
National Center for Fair & Open Testing

for further information:
Bob Schaeffer: (239) 395-6773
mobile: (239) 699-0468
for immediate release May 7, 2014
STAGNANT GRADE 12 NAEP RESULTS UNDERSCORE
FAILURE OF TEST-DRIVEN PUBLIC EDUCATION;
“NO CHILD LEFT BEHIND” AND STATE HIGH-STAKES EXAMS
DID NOT LEAD TO IMPROVED PERFORMANCE

Results from the National Assessment of Educational Progress (NAEP) for twelfth graders “underscore the failure of federal and state test-driven school policies,” according to the National Center for Fair & Open Testing (FairTest). Today’s report shows no NAEP score improvement for high school seniors in reading or math since 2009 and little progress over the past decade. Over the same period, performance gaps between racial groups have not narrowed significantly.
The NAEP trend is consistent with results from the ACT and SAT college admissions tests, where average scores continue to stagnate and racial gaps persist.
“How much more evidence do federal and state policy-makers need that driving schooling through standardized exams does not increase educational quality?” asked FairTest Public Education Director Bob Schaeffer. “It is time to abandon failed test-and-punish policies and adopt assessments that have been shown to improve teaching and learning.”
Schaeffer continued, “Across the country parents, teachers, students, and community activists are saying ‘enough is enough’ to testing misuse and overuse. They are opting out of tests, organizing protests against high-stakes exam overkill, and successfully pressing politicians to overhaul assessment policies. More elected officials must start listening to their constituents, who know what is going on in public school classrooms.” FairTest is a founder of the national Testing Resistance & Reform Spring campaign.

Professor Jack Hassard of Georgia State University concludes, after reviewing Tom Loveless’s report for Brookings, that the Common Core Standards have had little or no effect on NAEP math scores, as Loveles predicted a few years ago.

 

The states most aligned with CCSS had the smallest gains.

 

Overall, eighth grade math scores show very little improvement since the Common Core was rolled out in 2010.

 

He writes:

 

Between 1990 – 2013 there was a 22 point increase in 8th grade math. Over the 23 years this amounts to about a 1 point increase per year. However, the average score increase from 2009 – 2013, the years the Common Core has been used, has only increased 0.30 points per year, much less than before the roll out of the Common Core.

 

Well, four years is too soon to see the radical improvements that Bill Gates and others have promised. Maybe we will have to wait a full decade to know whether the billions spent on CCSS were well spent.

 

 

Follow

Get every new post delivered to your Inbox.

Join 150,295 other followers