Opinion: Reading gains are real

Earlier this month, Los Angeles Times columnist Michael Hiltzik declared Mississippi’s much touted gains in reading a sham. Hiltzik, whose most recent claim to fame was calling for unvaccinated covid deaths to be “mocked,” argued the Magnolia State “gamed” its results.

He was dead wrong. Embarrassingly so.

Relying on two bloggers’ statistical analysis, Hiltzik did his own mocking–excoriating members of the media for accepting at face value clear Mississippi gains on the National Assessment of Educational Progress (NAEP) reading test.

He wrote, “there is a lesson here, however. It’s not about how to effectively teach kids to read, but how difficult it is to teach journalists how to scrutinize statistical claims.”

On this point, Hiltzik and I agree. Because there were glaring red flags in the statistical analysis he bolstered in a floundering effort to disprove Mississippi’s achievements. Indeed, the bloggers who performed the analysis have since admitted they were wrong.

The Los Angeles Times has done no mea culpa for a column that was picked up across the country by other unquestioning outlets.

Before explaining the numerous flaws in Hiltzik’s botched takedown, let us back up and explore what he was attempting to take down.

Mississippi’s Very Real Gains

In 2013, the Mississippi Legislature passed a package of education reforms. Among the changes, the Literacy Based Promotion Act, a law which requires that students achieve basic grade level reading in third grade before being advanced to fourth grade.

Since the passage of the LBPA, sometimes referred to as the “Third Grade Reading Gate,” Mississippi students and teachers have performed admirably.

In 2013, white students in Mississippi ranked 49th in the country on the National Assessment of Educational Progress (NAEP) reading tests. Only Michigan, Wisconsin, Ohio, and Maine had worse scores for African American students.

By 2022, the latest year of NAEP testing, white fourth graders in Mississippi were performing 10th best among the states. Mississippi’s African American fourth graders rose to achieve the 5th highest score among the states.

These gains have been heralded by education experts and those reporting on education across the country, from EdPost to the Associated Press to the New York Times.

Hiltzik’s Hit

Michael Hiltzik could not allow Mississippi to get these accolades without a fight. It is clear from his writing that he loathes the very concept of our state. Half of his critique of Mississippi’s reading achievements is wholly unrelated to reading. Instead, he spends 10 solid paragraphs scoffing at Mississippi for having in place restrictions on abortion and failing to expand Medicaid.

Whatever a person’s views are on abortion and health care, it is illogical to use Mississippi’s current posture on those issues as proof that reading gains are not real.

Hiltzik’s next line of attack is to play the race card–highlighting a 25-point disparity between white students’ and black students’ scores in Mississippi.

(Hiltzik actually misreported the disparity as being 28 points, but who needs accuracy when lecturing about journalistic integrity?)

The existence of a racial gap is not proof that gains are not real, particularly when both races have significantly upped their game. Remember, Mississippi’s African American population swung from 5th worst in the nation to 5th best in the nation on NAEP’s fourth grade reading test between 2013 and 2022.

There, unfortunately, is a racial performance gap across the country. It’s not unique to us. Mississippi’s gap is actually the 7th lowest among states that track the statistic.

With a 37-point disparity, Hiltzik’s enlightened state of California is the second worst. It’s a national problem that deserves attention, but it does not render meaningless real gains made by both races in Mississippi.

Finally, Hiltzik turns to the only argument actually related to the claim that Mississippi “gamed” its gains. Two bloggers, Bob Somerby and Kevin Drum, undertook an analysis of the NAEP gains and came to the conclusion that the gains were almost all attributable to the lowest performing students in 3rd grade being held back, and thus, excluded from the 4th grade testing pool.

Their conclusion was that without retention, scores would not have gone up.

Had Hiltzik been the least bit curious, he could have spotted deficiencies in the argument from several miles away–even through Los Angeles’ dense smog.

For starters, Somerby and Drum rounded up the percentage of students being retained in the 3rd grade to 10 percent. No statistician would ever do that.

More importantly, they assumed for their analysis that the percentage of students being retained prior to the passage of the Literacy Based Promotion Act was zero. Any real statistician would have sought to understand what the retention rate was before the LBPA to have a point of comparison.

If, as is actually the case, the retention rate did not significantly increase from before the LBPA was passed, then the gains that have occurred since the LBPA cannot be attributed to retention.

In 2019, the Fordham Institute made the same mistake as Somerby and Drum, only to come back in August of 2022 and admit it was wrong. What the researchers at Fordham ultimately discovered was that the ages of Mississippi children taking the 4th grade NAEP prior to the LBPA and after the LBPA were the same.

In other words, the increased performance was not attributable to increasing the age of the testing pool. It was not a byproduct of holding students back.

The analysis relied upon by Hiltzik now includes a disclaimer at the top which reads:

“WARNING: This post is almost certainly wrong. Click here for an update showing how and why Mississippi test scores really did go up after 2013.”

The updated blog post draws the same conclusion drawn by the Fordham Institute.

It is also worth noting that even if retention had played a significant role in increasing performance, that would not really be a sign of a gamed system. A child held back does eventually move forward to the next grade. There aren’t a bunch of 18 year olds sitting in the third grade.

If giving a student an extra year to learn to read results in the pool of NAEP test takers performing better in the fourth grade, I struggle to understand how that is a bad thing. It seems like it would just be proof of the wisdom of the approach.

In-State Questions Raised

Over the weekend, the Mississippi Center for Public Policy released its own analysis of Mississippi’s reading gains. Focusing on the period between 2019 and 2022, the think tank concluded that “claims being made that there has been a ‘Mississippi miracle’ are not, sadly, substantiated by the facts.”

Mississippi Center’s CEO, Douglas Carswell, correctly notes that during this period, Mississippi’s fourth grade reading score declined from 219 points to 217 points.

So much of statistical analysis is wrapped up in start dates and end dates. Those education scholars who have sung Mississippi’s praise for improving reading scores use 2013 as the start date, since that is when the Literacy Based Promotion Act passed. The greatest period of growth in scores after the LBPA took place between 2013 and 2019.

Starting the analysis in 2019, at the current high water mark, does not account for the previous six year period of gains.

The fact that our state basically held serve between the 2019 test and the 2022 test should also not be discounted. NAEP testing typically takes place every other year. covid disrupted the testing pattern. 2022 was the first year since 2019 for the tests. Most of the nation experienced significant declines in scores.

In view of the systemic disruptions to learning caused by covid, Mississippi maintaining reading gains is an achievement, one that is reflected in NAEP rankings. In 2019, white students in Mississippi were ranked 18th nationally in fourth grade reading. That same year, African American students ranked 7th. In 2022, those rankings improved to 10th and 5th, respectively.

Where the Mississippi Center for Public Policy’s analysis warrants deeper conversation, in my estimation, is on questions outside of fourth grading reading achievements. It is possible both to recognize the tremendous gains made and simultaneously to recognize the need for more improvement in the system. It’s not a zero sum game.

Russ Latino is President of Empower Mississippi.