Archives for category: NAEP

Billy Townsend, Florida blogger, has reported regularly on Florida’s gaming of NAEP scores. He writes here that Governor Ron DeSantis is carrying out Jeb Bush’s old trick to inflate 4th grade NAEP scores. He calls the governor Ron Jebsantis. The trick is third grade retention, which ensures that the lowest scoring third graders never take the fourth grade NAEP test (the kids who take the NAEP test are selected at random).

Thus, DeSantis put out a flashy press release celebrating fourth grade NAEP scores in the test scores recently released. But, as usual, DeSantis neglects to mention the collapse of eighth grade NAEP scores. Somehow the kids who were retained in third grade managed to skip fourth grade and rejoin their classmates by eighth grade.

Here are his numbers, drawn from NAEP reports:

With that in mind, here is a view of Florida’s 2022 NAEP scores peaking in elementary school and dramatically worsening with the older cohorts —- which is ALL of the red numbers after the green baseline.

I personally put no stock in the twelfth grade numbers (which Billy extrapolated) because NAEP stopped testing seniors a decade ago. Seniors know that NAEP doesn’t count and they don’t do their best. Some don’t even try. Their answer sheets had doodles, or some just picked the (A) answer to every question or some were blank.

But the stark drop from fourth grade to eighth grade says something’s fishy in Florida.

Since the two sets of NAEP scores were released recently, commentators have gone into a panic about “learning loss” and used the declines to promote their favorite reform: more of this, less of that. DeSantis even released a press release claiming falsely that Florida’s formula of ignoring the pandemic was just right (California stuck with the CDC guidelines and did at least as well, maybe better, than Florida, but Gavin Newsom did not issue a press release).

Jan Resseger has words of perspective that I sum up as: why are we surprised that learning was disrupted by the pandemic?

My question, having served on the NAEP board for seven years, is why the media and the reform crowd thinks that NAEP scores should go up every year? Why should fourth and eighth graders this year know more than fourth and eighth graders two years ago or four years ago? Isn’t it reasonable to assume that students of the same age and grade are likely to have the same scores? Yet if they do, the media sends out loud lamentations that scores are “flat.” Oh, woe! Surely we want to see a rise in the scores of the lowest scoring students, and a narrowing of gaps, but the media assumes that everyone must increase their scores or the education system is failing. This is nuts. There is little or no relationship between the test scores of students in fourth and eighth grades and the economy of the future.

Jan Resseger writes:

Are the new National Assessment of Educational Progress scores a catastrophic indication that the U.S. public schools have fallen into decline? I don’t think so.

Early this week, the National Center for Education Statistics (NCES) released a large data set from National Assessment of Educational Progress exams administered last spring to 4th and 8th grade students in U.S. public schools. Last month, NCES released scores from tests administered to a smaller group of 4th graders. Both sets of scores show that the COVID pandemic seriously disrupted schooling for the nation’s children and adolescents.

Chalkbeat’s Matt Barnum explainswhat the National Assessment of Educational Progress (NAEP) is: “The National Assessment of Educational Progress (NAEP, pronounced nape) is a test administered by an arm of the U.S. Department of Education. It’s given periodically to a representative subset of American students in math and reading in grades four and eight. Scores are broken down by state and for a select handful of cities, too. The latest results are based on tests given between January and March 2022. The previous test was given in 2019, before the pandemic… Scores from a separate NAEP exam that has been given to 9-year-olds for many decades were previously released in September.”

The NAEP scores released this week were precipitously lower than scores on the NAEP when it was administered in 2019, before COVID—particularly in 8th grade math. The Washington Post’s Laura Meckler reports: “The portion of eighth-graders rated proficient or better in math fell to 27 percent, from 34 percent in 2019… the steepest decline in more than a half century of testing.” (The fact that every year relatively few students reach NAEP’s proficient level overall is because the NAEP “proficient” cut score is set artificially high; it marks what most people would define as “advanced.”)

Some people assume that this year’s drop in NAEP scores signals a reversal of progress, the beginning of a downward spiral. Others are using the scores as evidence for their particular reform or as evidence that their state had a better policy on school closures than other states. Meckler writes: “Partisans on all sides of the education debate seized on the results to advance competing ideas about the way ahead… The test results also offered fodder for those who argue bringing students back to campuses quickly was the right move… ‘We kept schools open in 2020, and today’s NAEP results once again prove we made the right decision,’ Florida Gov. Ron DeSantis (R) said on Twitter. But the data did not establish a connection between back-to-school policies and academic performance. In California, for instance, many public schools were closed well into the 2020-21 school year and some students never saw a classroom that year. But the declines were similar to those in Texas and Florida, where schools were ordered to reopen much sooner.”

In a blog post last month when the first set of 4th grade NAEP scores was released, I shared my own assessment of what had happened. I think the scores released last month and the scores released this week show the same thing. Here is some of what I said in that post.

***

There is no cause for panic. Schooling was utterly disrupted for the nation’s children and adolescents, just as all of our lives were interrupted in so many immeasurable ways. During COVID, while some of us have experienced the catastrophic death of loved ones, all of us have experienced less definable losses—things we cannot name.

I think this year’s NAEP scores—considerably lower than pre-pandemic scores—should be understood as a marker that helps us define the magnitude of the disruption for our children during this time of COVID. The losses are academic, emotional, and social, and they all make learning harder.

Schools shut down and began remote instruction in the spring of 2020, and many stayed online through the first half of last school year. While most public schools were up and running by last spring, there have been a lot of problems—with more absences, fighting and disruption, and overwhelming stress for educators. It is clear from the disparities in the scores released last week among high and low achievers that the disruption meant very different things to different children. It is also evident that the pandemic was a jolting shock to our society’s largest civic institution. It should be no surprise, then, that the attempt to get school back on track was so rocky all through last spring…

While the NAEP is traditionally used to gauge the trajectory of overall educational achievement over time, and while the trajectory has been moderately positive over the decades, the results released last week cannot by any means be interpreted to mean a change of the overall direction of educational achievement.

Education Week’s Sarah Schwartz asked Stanford University professor Sean Reardon (whose research tracks the connection of poverty and race to educational achievement) whether “it will take another 20 years to raise scores once again.” Reardon responded: “That’s the wrong question…. The question is: What’s going to happen for these (9-year-old) kids over the next years of their lives.” Schwartz describes more of Reardon’s response: “Children born now will, hopefully, attend school without the kinds of major, national disruptions that children who were in school during the pandemic faced. Most likely, scores for 9-year-olds, will be back to normal relatively soon, Reardon said. Instead, he said, we should look to future scores for 13-year-olds, which will present a better sense of how much ground these current students have gained.”

Schwartz reports: “Students at all levels lost ground during the past two years, but lower-performing students saw the biggest drops.” The test does not in any way measure the factors that contributed to the drop in scores for students who were already struggling, but the results shouldn’t be surprising. Some children live in families with internet access and enough computers that each of several children in the family could access online instruction simultaneously, while other children’s parents had to drive them to public library or fast food outlet parking lots to find any internet access at all. Some parents had sufficient time at home to supervise children and provide assistance during online instruction, while in other families, older siblings supervised younger siblings while trying to participate themselves in online instruction. Some children and adolescents simply checked out and neglected to log-on.

H. Hurley, a reader of the blog, left the following comment, which places NAEP hysteria into context:

The cherry on the journalistic cup cake related to recent NAEP reporting was an interview by Stephanie Ruhle on her 11:00 pm MSNBC program where she rushed in, of all people, ARNE DUNCAN, to discuss the CRISIS OF THE DROPPING NAEP SCORES. Her URGENCY in her set-up and interview was almost reported as a 3 alarm fire. Poor Arne. He actually tried to calm her reactions. But her hysteria is typical related to student test scores.


Nuts!


It’s obvious to real educators that a pandemic, million COVID deaths, ZOOM schooling, kids alone at home, banning books, masking, vaccing…anti vaccing, limited computer/Internet access, Jan6, school shootings, politics, chaos everywhere….shall we go on?


On top of this craziness, when children are finally returning to school, we TEST. We test & react in horror that children didn’t know the grade level content or skills. Scores dropped….who knew? Who could have predicted that?


ACTUALLY…….Anybody with some sense!
Children living in war, migration, fleeing, homeless, famine, rising fascism, massive crime, poverty, lead poisoning, hunger, job losses, craziness, etc…..are then tested under the WORST CONDITIONS.
Meanwhile, journalists hold up those results as if our children were living under heat lamps in incubators to be educated under the best conditions.


Stop the testing madness, end poverty, stop the political madness, allow families to raise their children with proper wages, fund schools, stop destroying public schools & use the election spending zillion$ on real people for a healthy nation.


My 2¢ worth!

Peter Greene wants to save time for all organizations that react to the latest NAEP scores. His press release works whether scores are up, down, or flat.

He writes:

It’s time once again to greet the release of another set of data from the NAEP testing machine, which means everyone is warming up their Hot Take generator. But if, like me, you’re getting tired of writing a response to the latest NAEPery, here’s a handy news release that will let you mad lib your way to NAEPy wisdom.


The new scores from the National Assessment of Education Progress (NAEP), known as The Nation’s Report Card, have been released, providing important data about [insert your preferred education policy area]. The recent crisis in [select your favorite policy-adjacent crisis] has clearly created a burgeoning issue of [select whatever Bad Thing you feel will most scare your audience in the direction of your preferred policy].

Says [head of your organization], “The new scores provide important evidence that now is the time for [insert whatever policy action your group always supports]. Clearly the [rise/drop/stagnation] in scores among [whichever subgroup cherry picking best suits your point] proves exactly what we have been arguing for [however long you’ve been at this.]”

[Insert paragraph of data carefully selected and crunched for your purposes. Add a graph if you like. People really dig graphs.]

“This is a clear indication,” says [your favorite go-to education expert], “that it is long past time to [do that thing your organization has been trying to get people to do for years]. Clearly [our preferred solution] is needed.” [Insert further sales pitch here as needed.]

You can expand on this if you wish, but make sure that you definitely do not–

* provide context for the data that you include

* offer perspective from NAEP’s many critics

* absolutely never ever reference the fact that the NAEP folks are extraordinarily clear that folks should not try to suggest a causal relationship between scores and anything else.

As always, the main lesson of NAEP is that contrary to the expectations of so many policy wonks, cold hard data does not actually solve a thing.

The NAEP remains a data-rich Rorschach test that tells us far more about the people interpreting the data than it does about the people from whom the data was collected. Button up your overcoat, prepare for greater-than-usual pearl-clutching and solution-pitching from all the folks who still think the pandemic shutdown is a great opportunity to do [whatever it is they have already been trying to do].

The National Assessment of Educational Progress released its report on student progress in math and reading for grades 4 and 8. As expected, scores plummeted due to the many disruptions in students’ lives during the pandemic. Schools closed, then opened. Some never closed. Some were online. Some were both in-person and online. Some students lost family members to the COVID. Some students had COVID. Some teachers died or got COVID. Many cities and towns closed down. There was not a right way or a wrong way. There were many people trying to figure out what was the right thing to do. It’s still not clear, although I personally think that vaccines and masks saved many lives and reduced the seriousness of the disease for those who got it.

Leonie Haimson reviewed the results here. She provides links to other cities and states.

The Commissioner of the National Center for Education Statistics, Peggy Carr, said this:

“There’s nothing in this data that tells us that there is a measurable difference in the performance between states and districts based solely on how long schools were closed,” NCES Commissioner Peggy Carr said.

I knew Peggy Carr when I served in the US Department of Education in the early 1990s. She is a career official and a straight-talker.

Jan Resseger is consistently the voice of wisdom on anything related to children and young people. In this post, she explains why we should not be panicked by the decline of NAEP scores. The scores reflected the toll that the pandemic exacted. But now that children are back in school, we can expect learning to proceed without major disruption.

She writes:

I think this year’s NAEP scores—considerably lower than pre-pandemic scores—should be understood as a marker that helps us define the magnitude of the disruption for our children during this time of COVID. The losses are academic, emotional, and social, and they all make learning harder….

Education Week’s Sarah Schwartz asked Stanford University professor Sean Reardon (whose research tracks the connection of poverty and race to educational achievement) whether “it will take another 20 years to raise scores once again.” Reardon responded: “That’s the wrong question…. The question is: What’s going to happen for these (9-year-old) kids over the next years of their lives.” Schwartz describes more of Reardon’s response: “Children born now will, hopefully, attend school without the kinds of major, national disruptions that children who were in school during the pandemic faced. Most likely, scores for 9-year-olds, will be back to normal relatively soon, Reardon said. Instead, he said, we should look to future scores for 13-year-olds, which will present a better sense of how much ground these current students have gained.”

Leonie Haimson looked closely at the score declines on the National Assessment of Educational Progress and was disappointed to see the outpouring of false prescriptions. She was critical of claims that students needed to make up for lost time by being subjected to longer school days and weeks.

The best response, she argues, based on years of research, is to reduce class size and give students the attention and care they need to make up for lost time.

Every major newspaper carried a story this morning about the sharp decline in NAEP scores because of the pandemic.

The moral of the story is that students need to have human contact with a teacher and classmates to learn best. Virtual learning is a fourth-rate substitute for a real teacher and interaction with peers.

Tech companies have told us for years that we should reinvent education by replacing teachers with computers. We now know: Virtual learning is a disaster.

The crisis we should worry about most is the loss of experienced teachers, who quit because of poor working conditions, low pay, and attacks by “reformers” who blame teachers at every opportunity.

The pandemic isolated children from their teachers. It caused them to be stuck in front of a computer. They were bored.

They needed human interaction. They needed to look into the eyes of a teacher who encouraged them to do better, a teacher who explained what they didn’t understand.

The NAEP scores are a wake-up call. We must treasure our teachers and recognize the vital role they play in educating the next generation.

Any politician who disrespects teachers by calling them “pedophiles” and “groomers” should be voted out of office.

Every “reformer” who disparages teachers should be required to teach for one month, under close supervision, of course.

James Harvey recently retired as executive director of the National Superintendents’ Roundtable. He is a member of the board of the Network for Public Education. In this post, he describes how the benchmarks used by the National Assessment of Educational Progress are misused to attack American education. The “achievement levels” were created in 1990 when Chester Finn Jr., an enemy of public schools, was chair of the National Assessment Governing Board. They were designed to make American student achievement look worse than it was. The media and the public think that “proficient” means “grade level.” It does not. It is equivalent to a solid A. Yet how many hundreds or thousands of times (e.g. the charter propaganda film “Waiting for Superman”) have you been told that most American students score “below grade level”? It’s not true. To be blunt, it’s a lie.

James Harvey wrote on Valerie Strauss’s “Answer Sheet” blog at The Washington Post:

Every couple of years, public alarm spikes over reports that only one-third of American students are performing at grade level in reading and math. No matter the grade — fourth, eighth or 12th — these reports claim that tests designed by the federal government, the National Assessment of Educational Progress (NAEP), demonstrate that our kids can’t walk and chew gum at the same time. It’s nonsense.


In fact, digging into the data on NAEP’s website reveals, for example, that 81 percent of American fourth-graders are performing at grade level in mathematics. Reading? Sixty-six percent. How could this one-third distortion come to be so widely accepted? Through a phenomenon that Humpty Dumpty described best to Alice in “Through the Looking Glass”: “When I use a word it means just what I choose it to mean.”


Here, the part of Humpty Dumpty was played by Reagan-era political appointees to a policy board overseeing NAEP. The members of the National Assessment Governing Board, most with almost no grounding in statistics, chose to define the term “proficient” as a desirable goal in the face of expert opinion that such a goal was “indefensible.”

Here’s a typical account from the New York Times in 2019 reporting on something that is accurate as far as it goes: results from NAEP indicate that only about one-third of fourth- and eighth-graders are “proficient” in reading.


But that statement quickly turns into the misleading claim that only one-third of American students are on grade level. The 74, for example, obtained $4 million from the Walton and DeVos foundations in 2015 by insisting that “less than half of our students can read or do math at grade-level.”


The claim rests on a careless conflation of NAEP’s “proficient” benchmark with grade-level performance. The NAEP assessment sorts student scores into three achievement levels — basic, proficient, and advanced. The terms are mushy and imprecise. Still, there’s no doubt that the federal test makers who designed NAEP see “proficient” as the desirable standard, what they like to describe as “aspirational.”


However, as Peggy Carr from the National Center for Education Statistics, which funds NAEP, has said repeatedly, if people want to know how many students are performing at grade level, they should be looking at the “basic” benchmark. By that logic, students at grade level would be all those at the basic level or above, which is to say that grade-level performance in reading and mathematics in grades 4, 8 and 12, is almost never below 60 percent and reaches as high as 81 percent.
And the damage doesn’t stop with NAEP. State assessments linked to NAEP’s benchmarks amplify this absurd claim annually, state by state.

While there’s plenty to be concerned about in the NAEP results, anxiety about the findings should focus on the inequities they reveal, not the proportion of students who are “proficient.”
Considering the expenditure of more than a billion dollars on NAEP over 50-odd years, one would expect that NAEP could defend its benchmarks by pointing to rock-solid studies of their validity and the science behind them. It cannot.


Instead, the department has spent the better part of 30 years fending off a scientific consensus that the benchmarks are absurd. Indeed, the science behind these benchmarks is so weak that Congress insists that every NAEP report include the following disclaimer: “[The Department of Education] has determined that NAEP achievement levels should continue to be used on a trial basis and should be interpreted with caution” (emphasis added).


Criticisms of the NAEP achievement levels
What is striking in reviewing the history of NAEP is how easily its policy board has shrugged off criticisms about the standards-setting process. The critics constitute a roll call of the statistical establishment’s heavyweights. Criticisms from the likes of the National Academy of Education, the Government Accounting Office, the National Academy of Sciences, and the Brookings Institution have issued scorching complaints that the benchmark-setting processes were “fundamentally flawed,” “indefensible,” and “of doubtful validity,” while producing “results that are not believable.”
How unbelievable? Fully half the 17-year-olds maligned as being just basic by NAEP obtained four-year college degrees. About one-third of Advanced Placement Calculus students, the crème de la crème of American high school students, failed to meet the NAEP proficiency benchmark. While only one-third of American fourth-graders are said to be proficient in reading by NAEP, international assessments of fourth-grade reading judged American students to rank as high as No. 2 in the world.

For the most part, such pointed criticism from assessment experts has been greeted with silence from NAEP’s policy board.


Proficient doesn’t mean proficient


Oddly, NAEP’s definition of proficiency has little or nothing to do with proficiency as most people understand the term. NAEP experts think of NAEP’s standard as “aspirational.” In 2001, two experts associated with NAGB made it clear that:
“[T]he proficient achievement level does not refer to “at grade” performance. Nor is performance at the Proficient level synonymous with ‘proficiency’ in the subject. That is, students who may be considered proficient in a subject, given the common usage of the term, might not satisfy the requirements for performance at the NAEP achievement level.”

Lewis Carroll’s insight into Humpty Dumpty’s hubris leads ineluctably to George Orwell’s observation that “[T]he slovenliness of our language makes it easier for us to have foolish thoughts.”

NAEP and international assessments


NAEP’s proficiency benchmark might be more convincing if most students abroad could handily meet it. That case cannot be made. Sophisticated analyses between 2007 and 2019 demonstrate that not a single nation can demonstrate that even 50 percent of its students can clear the proficiency benchmark in fourth-grade reading, while only three could do so in eighth-grade math and one in eighth-grade science. NAEP’s “aspirational” benchmark is pie-in-the-sky on a truly global scale.
What to do?

NAEP is widely understood to be the “gold standard” in large-scale assessments. That appellation applies to the technical qualities of the assessment (sampling, questionnaire development, quality control and the like) not to the benchmarks. It is important to say that the problem with NAEP doesn’t lie in the assessments themselves, the students, or the schools. The fault lies in the peculiar definition of proficiency applied after the fact to the results.

Here are three simple things that could help fix the problem:


• The Department of Education should simply rename the NAEP benchmarks as low, intermediate, high, and advanced.

• The department should insist that the congressional demand that these benchmarks are to be used on a trial basis and interpreted with caution should figure prominently, not obscurely, in NAEP publications and on its website.

• States should revisit the decision to tie their “college readiness” standards to NAEP’s proficiency or advanced benchmarks. (They should also stop pretending they can identify whether fourth-graders are “on track” to be “college ready.”)

The truth is that NAEP governing board lets down the American people by laying the foundation for this confusion. In doing so, board members help undermine faith in our government, already under attack for promoting “fake news.” The “fake news” here is that only one-third of American kids are performing at grade level.

It’s time the Department of Education made a serious effort to stamp out that falsehood.

Paul Bonner, who recently retired as a principal in Alabama, wrote the following comment as part of a discussion of administering NAEP to kindergartners.

He wrote:

One of the experiences that made me aware that my time with public education was coming to an end was when our district began testing kindergartners. I would walk into kindergarten classrooms and watch students struggle and often cry over the inability to navigate iPads. I would leave those classrooms shaken to the core. The students who could work with the devices were not making decisions about correct answers but through simply getting the program to move from question to question. Almost none of these students could understand what the test was asking them to do. This angered me significantly because what we were focusing on ignored the activities that were needed to build an actual foundational developmental standard. No focus on gross and fine motor skill development or social and emotional growth. No test below third grade will give us meaningful understanding of what children actually know and that really is beside the point. The poor quality of most of the tests I have seen keep us from understanding what those form third grade through twelve understand! What we are doing to children, or being asked to do, is criminal and a denial of how the brain can get to a point of meaningful inquiry. The fact that people who have no experience with child development and have done no meaningful study of the early brain, provides further evidence that our society and polity has no appreciation for the professional approach required to raise children to become successful adults. It just seems to be getting worse. I am absolutely appalled to see another presidential administration and the plethora of state governments that refuse to see the damage they are doing. This predatory capitalism that has so infected education, and all of governance, just might result in the same effect led poisoning had on Rome.