The National Assessment Governing Board, the federal agency in charge of the NAEP assessments, is aware that the achievement levels (Basic, Proficient, Advanced) are being misused. They are considering tinkering with the definitions of the levels. NAGB has invited the public to express its views. Below is my letter. If you want to weigh in, please write to NAEPALSpolicy@ed.gov and Peggy.Carr@ed.gov. Responses must be received by September 30.

My letter:

Dear NAEP Achievement-Level-Setting Program,

As a former member of the National Assessment Governing Board, I am keenly interested in the improvement and credibility of the NAEP program.

I am writing to express my strong support for a complete rethinking of the NAEP “achievement levels.” I urge the National Assessment Governing Board to abandon the achievement levels, because they are technically unsound and utterly confusing to the public and the media. They serve no purpose other than to mislead the public about the condition of American education.

The achievement levels were adopted in 1992 for political reasons: to make the schools look bad, to convey simplistically to the media and the public that “our schools are failing.”

The public has never understood the levels. The media and prominent public figures regularly report that any proportion of students who score below “NAEP proficient” is failing, which is absurd. The two Common Core-aligned tests (PARCC and SBAC) adopted “NAEP Proficient” as their passing marks, and the majority of students in every state that use these tests have allegedly “failed,” because the passing mark is out of reach, as it will always be.

The National Center for Educational Statistics (NCES) has stated clearly that “Proficient is not synonymous with grade level performance.” Nonetheless, public figures like Michelle Rhee (who was chancellor of the DC public schools) and Campbell Brown (founder of the website “The 74”) have publicly claimed that the proficiency standard of NAEP is the bar that ALL students should attain. They have publicly stated that American public education is a failure because there are many students who have not reached NAEP proficient.

In reality, there is only one state in the nation–Massachusetts–where as much as 50% of students have attained NAEP Proficient. No state has reached 100% proficient, and no state ever will.

When I served on NAGB for seven years, the board understood very well that proficient was a high bar, not a pass-fail mark. No member of the board or the staff expected that some day all students would attain “NAEP Proficient.” Yet critics and newspaper consistently use NAEP proficient as an indicator that “all students” should one day reach. This misperception has been magnified by the No Child Left Behind Act, which declared in law that all students should be “proficient” by the year 2014.

Schools have been closed, and teachers and principals have been fired and lost their careers and their reputations because their students were not on track to reach an impossible goal.

As you well know, panels of technical experts over the years have warned that the achievement levels were not technically sound, and that in fact, they are “fatally flawed.” They continue to be “fatally flawed.” They cannot be fixed because they are in fact arbitrary and capricious. The standards and the process for setting them have been criticized by the General Accounting Office, the National Academy of Sciences, and expert psychometricians.

Whether using the Angoff Method or the Bookmarking Method or any other method, there is no way to set achievement levels that are sound, valid, reliable, and reasonable. If the public knew that the standards are set by laypersons using their “best judgment,” they would understand that the standards are arbitrary. It is time to admit that the standard-setting method lacks any scientific validity.

When they were instituted in 1992, their alleged purpose was to make NAEP results comprehensible to the general public. They have had the opposite effect. They have utterly confused the public and presented a false picture of the condition and progress of American education.

As you know, when Congress approved the achievement levels in 1992, they were considered experimental. They have never been approved by Congress, because of the many critiques of their validity by respected authorities.

My strong recommendation is that the board acknowledge the fatally flawed nature of achievement levels. They should be abolished as a failed experiment.

NAGB should use scale scores as the only valid means of conveying accurate information about the results of NAEP assessments.

Thank you for your consideration,

Diane Ravitch
NAGB, 1997-2004
Ph.D.
New York University

ALSO:

The National Superintendents Roundtable wrote a letter.

I urge you to read this here.

The letter documents the many scholarly studies criticizing the NAEP achievement levels.

Here is an excerpt:

“NAGB hired a team of evaluators in 1990 to study the process involved in developing the three levels. A year later the evaluators were fired after their draft report concluded that the process “must be viewed as insufficiently tested and validated, politically dominated, and of questionable credibility.”

“In 1993, the U.S. General Accounting Office labeled the standard-setting process as “procedurally flawed” producing results of “doubtful accuracy.”

“In 1999, the National Academy of Sciences reported the achievement-level setting procedures were flawed: “difficult and confusing . . . internally inconsistent . . . validity evidence for the cut scores is lacking . . . and the process has produced unreasonable results.”

“Shortly after No Child Left Behind was signed into law in 2001, Robert Linn, past president of the American Educational Association and of the National Council on Measurement in Education, and former editor of the Journal of Educational Measurement called the “target of 100% proficient or above according to the NAEP standards is more like wishful thinking than a realistic possibility.”

“In 2007, researchers concluded that fully a third of high school seniors who completed calculus, the best students with the best teachers in the country, could not clear the proficiency bar. Moreover, they added, fully 50 percent of those who scored “basic” in twelfth grade math had achieved a bachelor’s degree (a proportion comparing favorably with four-year degree rates at public universities).

“The Buros Institute, named after the father of Mental Measurements Yearbook, criticized the lack of a validity framework for NAEP assessment scores in 2009 and recommending continuing “to explore achievement level methodologies.”

“Fully 30 percent of 12th-graders who completed calculus were deemed to be less than proficient, said a Brookings Institution scholar in 2016, a figure that jumped to 69 percent for pre-calculus students and 92 percent for students who completed trigonometry and Algebra I. These data “defy reason” and “refute common sense,” he concluded.

“Finally, the NAS study to which the proposed rule responds took note in 2016 of the “controversy and disagreement around the achievement levels, noting that Congress has insisted since 1994 that the achievement levels are to be used on a trial basis until on objective evaluation determined them to be “reasonable, reliable, valid, and informative to the public.”

“In the Roundtable’s judgment, such an objective evaluation has yet to be completed and a determination that the achievement levels are “reasonable, reliable, valid, and informative to the public” has yet to be seen.

“Linking studies conclude most students in most nations cannot clear “proficiency” bar

“The Roundtable points also to research studies dating from 2007 to 2018 indicating NAEP’s proficiency bar is beyond the reach of most students in most nations. When Gary Phillips of the American Institutes of Research (and former Acting Commissioner of NCES) asked how students in other nations would perform if their international assessment results were expressed in terms of NAEP achievement levels, his results were sobering. The results demonstrated that just three nations (Singapore, the Republic of Korea, and Japan) would have a majority of their students clear the NAEP bar in 8th-grade mathematics, while Singapore alone could meet that standard (more than 50% of students clearing the bar) in science.

“Subsequently Hambleton, Sireci, and Smith (2007) and also Lim and Sireci (2017) reached conclusions similar to those of Phillips.”

The fact is that “NAEP proficiency” is an impossible goal for most students. To recognize that does not lower standards. It acknowledges common sense.

Not every runner will ever run a four-minute mile. Some will. Most wont.