Conflict of Interest Disclosure

This project involves defining criteria to conduct an evaluation of the extent to which the achivement levels that are used to report NAEP results are reliable, valid, reasonable, and informative, and then using those criteria to guide data collection, weigh the evidence, and draw conclusions. All of the information that the committee gathers will pertain to test scores and communicating test results; some will be quantitative and technical, and other information will be qualitative.

A wide range of expertise is needed to adequately address the charge. We are nominating individuals with the following areas of expertise: program evaluation in education and experience using both qualitative and quantitative data; educational measurement, particularly in the areas of setting standards and developing score reports as well as familiarity with the psychometrics behind NAEP; education policy, particularly policy uses of NAEP results; research methodology and statistics as it pertains to the more technical aspects of NAEP and setting standards; equity and fairness for special populations (students who have a disability, are English language learners, are racial/ethnic minorities, or are disadvantaged) as it pertains to the process of setting standards and reporting results using achievement levels; and the subject matter of reading and mathematics. Some of the nominated individuals have expertise that spans more than one cateogry, and together the individuals we have nominated have all of the needed expertise.

Currently NAEP results are reported using three achievement levels: “basic,” “proficient,” and “advanced.” Several evaluations of these achievement levels were conducted in the 1990s; most were critical of the process for setting these standards, and the cut scores (the score that separates one achievement level from the next) it produced. These criticisms provoked sharp disagreement in the measurement field, with many professionals making their perspectives known in public statements. These disagreements have softened over time, but many individuals with extensive measurement and standard setting expertise still hold strong opinions. In selecting individuals for the committee, we have tried to avoid those who would come into the project with fixed opinions about the achivement levels that are used in NAEP.

Meeting this goal was complicated by the fact that individuals with expertise in setting standards are in high demand and already working in ways that would present a conflict of interest. Some of these individuals work for testing companies that favor a certain method for setting cut scores. Some serve on technical panels designed to provide advice to NAEP about their standard setting practices. Others were directly involved in designing/implementing the procedures that we will be evaluating. We have carefully vetted our nominees to ensure that they will not have these conflicts of interest.

We have sought to balance the committee along gender and demographic characteristics, as well as prior NRC experience. Seven of the 15 committee members are female; the chair and one other member are African American; and one member is Latino. Four of the committee members—including the chair—have been members of BOTA or the DBASSE Advisory Committee, and eight have served on multiple NRC committees; three nominees have not previously served on NRC committees. Together, the nominees represent nine states and all regions of the country.