NAEP data is more reliable than PARCC or SBAC

TruthThere has been a lot of buzz the last few days about the release of the release of the Nation’s Report Card. Each year, the National Assessment of Educational Progress (NAEP) collects a random sample of students for testing that represents, not only each state, but also, demographic subsets within each state. The tests are then scored and used to compare the educational progress of states against other states and the national average. Although the test has undergone revisions and updates, the NAEP has been doing this since the early ’60s; however, readily accessible data is only available back to the early 90s.

Much of the media has been reporting Louisiana as a dismal failure, and the pro-education reformers like to twist the data, and depending on what they see, either tout the successes of Common Core, or express how much we need Common Core. Some have even reported that the new PARCC and SBAC tests will provide more accurate feedback. That is absolutely incorrect. Here is why.

In criterion-referenced and norm-referenced tests, panels of knowledgeable participants reviewing and discuss the standards tested and decide on a minimum body of knowledge in each subject at each grade level. Then cut scores are set to determine the minimum scaled score required to meet each achievement level; usually Basic, Proficient and Mastery. Notice that this process takes place before the test is administered. This gives NAEP the ability to compare each and every participant to the same standard. By contrast, both the SBAC and PARCC cut scores were determined after the raw scores were available. In addition, the superintendent of each state has the ability to set their own cut scores. In fact, in PARCC, only one state used the “agreed upon” cut scores by the PARCC consortium. The NAEP test not been revised in a number of years, and the same cut scores have been used since the last revision. These facts illustrate that there is no possible way to compare PARCC and SBAC results other states in their respective consortium, or to the NAEP results. For more information on the NAEP cut score process, click here.

Since we have determined that the NAEP results are far more accurate than the two consortium assessments, we decided to investigate some of the results and compare them to the claims made about Common Core and other education reform efforts. The following charts reveal the findings. Since achievement levels are complex and don’t give a clear picture of actual performance, we have analyzed scaled scores. The details are captioned beneath each graphic.

AvgScaledScore4M

(Above) In this first graphic, we compare Louisiana’s average scaled score in 4th grade Math to the average over a number of years.

AvgScaledScore8M

(Above) Louisiana’s average scaled score in 8th grade Math compared to the national average.

AvgScaledScore4R

(Above) Louisiana’s average scaled score in 4th grade Reading compared to the national average.

AvgScaledScore8R

(Above) Louisiana’s average scaled score in 8th grade Reading compared to the national average.

PARCCstates

(Above) In this graphic, we compare Louisiana’s scores in 4th & 8th grade Math and 4th & 8th grade Reading to the national score AND the scores of states that began in the PARCC consortium. Note: Many of the states have exited the PARCC consortium.

In the first graphics, we simply made comparisons between Louisiana and the national average using the results of the most recent assessment. In reality, a pro-reformer could argue that Louisiana is in bad shape compared to the nation, and that Common Core and PARRC are needed to improve our students’ achievement. You could use the same argument in the next graphic when compared to other PARCC states. This would not be a valid argument. Another argument that reformers like to make is “all children can learn regardless of zipcode.” Well, we don’t disagree, but the fact is, these assessments don’t measure ability to learn. They measure achievement, and achievement is affected by so many factors that it isn’t possible to name them all. Reformers like to use “free/reduced lunch eligibility” as a measurement of poverty. Access to technology, family structure, health care, community involvement and absentee parents all contribute to self-confidence, self-motivation and self-discipline are all things that contribute directly to achievement. Using their preferred terminology, we illustrate “poverty” below.

PARCClunch2015

(Above) This is where the comparisons get interesting. In looking at the scaled scores of students eligible for free/reduced lunch vs student who do not, it is pretty clear that not only in the PARCC states, but nationally, there is a huge performance gap between the two.

NationalGAP

(Above) Here is an illustration of what the national performance gap looks like over time as it relates to poverty.

LouisianaGAP

(Above) Last, but certainly not least, we illustrate the Louisiana performance gap over time as it relates to poverty.

Now that you’ve seen this data illustrated, we want to point out a couple of things. Looking at the last two graphics illustrating gap, you can see that somewhere around 2002, the gap closed slightly, but with the exception of that one change, the gap remains pretty constant throughout the years. What is significant about 2002? In January 2002, the No Child Left Behind Act (NCLB) went into effect. While initially, the gap moved slightly, from that point on, there is no reduction worthy of note. Lest we remind you, this was the point of NCLB; to close the gap. Look closely at the difference between 2013-2015. This is when Common Core was implemented in almost all of the states. The gap actually increased. What was the point of Common Core and RTTT? To close the gap.

After looking at this data, and digesting what they mean, what conclusions can you draw? Here are ours.

  1. No amount of money dedicated to assessment, accountability will close the performance gap.
  2. Identifying schools as failures and blaming teachers for the outcomes will not closed the performance gap.
  3. Allowing private corporations to convert tax dollars to profit in the name of “closing the gap,” isn’t happening.
  4. Until the powers that be address poverty, and other factors outside of the classroom, the gap will remain.