Evaluation of National Assessment of Educational Progress Achievement Levels

Consensus

Download PDF

Paperback - $62.10
Ebook - $49.49

Since 1969, the National Assessment of Educational Progress (NAEP) has been providing policymakers, educators, and the public with reports on academic performance and progress of the nation's students. The assessment is given periodically in a variety of subjects: mathematics, reading, writing, science, the arts, civics, economics, geography, U.S. history, and technology and engineering literacy. NAEP is given to representative samples of students across the U.S. to assess the educational progress of the nation as a whole.

Since 1992, NAEP results have been reported in relation to three achievement levels: basic, proficient, and advanced. However, the use of achievement levels has provoked controversy and disagreement, and evaluators have identified numerous concerns. This publication evaluates the NAEP student achievement levels in reading and mathematics in grades 4, 8, and 12 to determine whether the achievement levels are reasonable, reliable, valid, and informative to the public, and recommends ways that the setting and use of achievement levels can be improved.

Publication information

288 pages · 6 x 9 · paperback
ISBN Paperback: 0-309-43817-9
ISBN Ebook: 0-309-43818-7
DOI: https://doi.org/10.17226/23409

Suggested citation

National Academies of Sciences, Engineering, and Medicine. 2017. Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press.

Import this citation to:

Christopher F. Edley, Jr.

Christopher Edley, Jr. (Chair) is the honorable William H. Orrick, Jr. distinguished professor, and faculty director at the Chief Justice Earl Warren Institute on Law and Social Policy at the University of California Berkeley School of Law. He was dean of the U.C. Berkeley School of Law from 2004 to 2013, after 23 years as a Harvard Law professor. His academic work is in administrative law, civil rights, education policy, and domestic public policy generally. Professor Edley has moved between academia and public service, each enriching the other and together giving him broad familiarity with many areas of public policy. He served in White House policy and budget positions under Presidents Jimmy Carter and Bill Clinton. In Clinton’s OMB, he oversaw budgets and legislative initiatives for five cabinet departments and over 40 independent agencies, with budget responsibility totaling in the hundreds of billions of dollars. As senior counsel to Clinton, he directed a government-wide review of affirmative action programs. Edley held senior positions in five presidential campaigns, including senior policy adviser for Barack Obama; he then served on Obama’s Transition Board with responsibility for Education, Immigration and Health. More recently, Edley co-chaired the congressionally chartered National Commission on Education Equity and Excellence (2011-2013). The Commission’s charge was to revisit the 1983 report, A Nation at Risk, and recommend future directions for reform; he chairs the follow-on effort, For Each & Every Child. He is a member of the American Academy of Arts & Sciences, National Academy of Public Administration, Council of Foreign Relations, Gates Foundation’s National Programs Advisory Panel, and has served on many panels for the National Research Council. He has served on numerous NRC panels, most recently the Committee on the Independent Evaluation of DC Public Schools, and the Committee on Incentives and Test-based Accountability. He served as co-chair of the Committee on Performance Levels for Adult Literacy. He holds a B.A. in mathematics from Swarthmore College, an M.A. from Harvard University John F. Kennedy School of Government, and a J.D. from Harvard Law School.

Peter Afflerbach

Peter Afflerbach is professor of reading in the Department of Teaching and Learning, Policy and Leadership at the University of Maryland. He began his teaching career as a Chapter One teacher in grades K-6, and he then taught remedial reading and writing in junior high school and high school English. Dr. Afflerbach’s research interests focus on reading assessment, reading comprehension strategies, and the verbal reporting methodology. His most recent work focuses on aspects of individual differences in reading development that are sometimes neglected in reading theory and practice, including motivation and engagement, metacognition, student self-efficacy and self-concept, and epistemic beliefs. Dr. Afflerbach has served for 14 years on the Standing Reading Committee of the National Assessment of Educational Progress. He was a member of the 2009 NAEP Reading Framework Committee and of the Feedback Committee for the Common Core State Standards/English Language Arts. He has served on numerous committees and panels for the Programme for International Student Assessment, National Assessment of Adult Literacy, and the National Accessible Reading Assessment Projects. Dr. Afflerbach has won numerous awards. His dissertation won both the International Reading Association’s Outstanding Dissertation Award, and the Literacy Research Association’s Outstanding Student Research Award. Dr. Afflerbach has earned awards from the American Educational Research Association for his classroom assessment research, and his reviewing in Educational Researcher. Peter was elected to the International Reading Association’s Reading Hall of Fame in 2009. He received his Ph.D. in reading psychology and his M.S. in developmental reading from the State University of New York at Albany.

Sybilla Beckmann

Sybilla Beckmann is Josiah Meigs distinguished teaching professor of mathematics at the University of Georgia. Previously she taught at Yale University as a J.W. Gibbs instructor of mathematics. Beckmann has done research in arithmetic geometry, but her current interests are mathematical cognition, the mathematical education of teachers, and mathematics content for PreK through Grade 8. She is interested in helping college faculty learn to teach mathematics content courses for elementary and middle grades teachers and she works with graduate students and postdoctoral fellows toward that end as director of the Mathematicians Educating Future Teachers (MEFT) program in the University of Georgia mathematics department. Beckmann was a member of the writing team of NCTM’s Curriculum Focal Points for Prekindergarten through Grade 8 Mathematics, was a member of Committee on Early Childhood Mathematics of the National Research Council and co-author of its report, Mathematics Learning in Early Childhood: Paths Toward Excellence and Equity, has worked on the development of several state mathematics standards, and was a member of the mathematics writing team for the Common Core State Standards for Mathematics. She was also a member of the NRC Committee Toward Integrated STEM Education: Developing a Research Agenda and is currently a member of the U.S. National Commission on Mathematics Instruction. Beckmann has won several teaching awards, including the Josiah Meigs Distinguished Teaching Professorship, which is the highest teaching honor at the University of Georgia, the Regents’ Teaching Award from the University System of Georgia, and the Louise Hay Award for Contributions to Mathematics Education awarded by the Association for Women in Mathematics. She has a Ph.D. in mathematics from the University of Pennsylvania.

H. Russell Bernard

H. Russell Bernard (NAS) is research professor of anthropology and director of the Institute for Social Science Research at Arizona State University. He is also professor emeritus of anthropology at the University of Florida. Bernard has done research in Greece, Mexico, and the United States and has taught or done research at universities in the U.S., Greece, Japan, and Germany. Bernard's areas of research include technology and social change, language death, and social network analysis. Since 1987, Bernard has participated in summer courses, sponsored by the U.S. National Science Foundation, on research methods and research design. Bernard was the 2003 recipient of the Franz Boas Award from the American Anthropological Association and is a member of the National Academy of Sciences. He is currently a member of the NRC International Temporary Nominating Group for Class V: Behavioral and Social Sciences and was previously a member of Committee to Review the State of Postdoctoral Experiences in Scientists and Engineers and Committee on the Use of Chimpanzees in Biomedical and Behavioral Research. He received his Ph.D. in anthropology from the University of Illinois at Urbana-Champagne.

Karla Egan

Karla Egan is an associate with the National Center for the Improvement of Educational Assessment where she provides technical support to national, state, and local education agencies on issues related to the design, development, implementation, and documentation of general assessments and alternate assessments. Dr. Egan’s areas of expertise include standard setting, achievement-level descriptors, and test security. She also has experience in developing achievement-level descriptions and setting standards for the alternative assessments given to students with significant learning disabilities. She has contributed chapters for edited books including Test Fraud: Statistical Detection and Methodology (with Smith, 2014; with Skorupski, 2014), Handbook of Research in Classroom Assessment (with Schneider & Julian, 2012), Setting Performance Standards: Foundations, Methods, and Innovations (with Schneider & Ferrara, 2012), and Accessible Tests of Student Achievement: Issues, Innovations, and Applications (with Schneider & Ferrara, 2011). During her career, Dr. Egan has designed and led over 40 standard setting workshops. She has implemented almost all major standard setting methods (e.g., Bookmark, Modified Angoff, Body of Work). She created a nationally recognized framework to develop achievement-level descriptors (Egan, Schneider, & Ferrara, 2012) and implemented the framework for the Smarter Balanced Assessment Consortium. Prior to joining the Center for Assessment, Dr. Egan was a senior research scientist for CTB/McGraw-Hill. Dr. Egan holds a Ph.D. in sociology from the University of Massachusetts, Amherst.

David J. Francis

David J. Francis is Hugh Roy and Lillie Cranz Cullen distinguished university chair and director of the Texas Institute for Measurement, Evaluation, and Statistics at the University of Houston. He also currently directs the Center for Research on Educational Achievement and Teaching of English Language Learners (CREATE), the ELL Strand of the Center on Instruction, and a large program of research on language and literacy development in Spanish-speaking children that is funded by NICHD and IES. Dr. Francis is a member of the Independent Review Panel for the National Assessment of Title I, a former Chairman of the Advisory Council on Education Statistics, and member of the Technical Advisory Group of the What Works Clearing House. He also served as a member of the National Literacy Panel for Language Minority Children and Youth and as a methodological consultant to the National Reading Panel. He is a frequent advisor to the Department of Education on statistical issues, assessment and accountability, and English Language Learners. Dr. Francis has collaborated in research on reading and reading disabilities, attention problems, developmental consequences of brain injuries and birth defects, and adolescent alcohol abuse. He is a recipient of the University of Houston Teaching Excellence Award and a former member of the National Institute of Health's Behavioral Medicine Study Section. He is a fellow of Division 5 (Measurement, Evaluation, and Statistics) of the American Psychology Association and He has been on numerous NRC committees and is currently chair of the Board on Testing and Assessment. In 2006 he received the Albert J. Harris Award from the International Reading Association, and he has also received the University of Houston’s Teaching Excellence Award and the Excellence in Research and Scholarship Award. He received the Esther Farfel Award in 2008, recognizing his career accomplishments in research, teaching, and service; it is the highest award given to faculty members at the University of Houston. He holds a B.S. in psychology from Kalamazoo College, and an M.A. and Ph.D. in clinical neuropsychology from the University of Houston.

Margaret E. Goertz

Margaret E. Goertz is a professor emerita of education policy and a senior researcher at the Consortium for Policy Research in Education in the Graduate School of Education at the University of Pennsylvania. She has over 35 years of experience conducting and leading national and state-level studies on education policy and policy implementation, including state and local implementation of Title I, NCLB and IDEA; the design and implementation of standards-based reform by state education agencies, school districts, and elementary and secondary schools; and the interface between federal and state accountability and school improvement policies. She recently completed a study of how state education agencies are organized to manage and use evidence in their policies or practices to improve low-performing schools. Dr. Goertz was a member of the NRC’s Committee on Understanding the Influence of Standards in Science, Mathematics and Technology Education, and the Committee on Goals 2000 and the Inclusion of Students with Disabilities. Prior to joining the University of Pennsylvania faculty, she taught at the Bloustein School of Planning and Public Policy at Rutgers University and was executive director of the Education Policy Research Division of Educational Testing Service. She holds a Ph.D. in social science from Syracuse University.

Laura Hamilton

Laura Hamilton is a senior behavioral scientist and research quality assurance manager at the RAND Corporation, a professor at the Pardee RAND Graduate School, and an adjunct faculty member in the University of Pittsburgh's Learning Sciences and Policy program. Her research addresses educational assessment, accountability, the measurement and evaluation of instruction and school leadership, the use of data for instructional decision making, and evaluation of technology-based curriculum reforms. She has led several large-multi-site studies and has expertise in the collection and analysis of interview, focus group, survey, and student outcome data. Recent projects include an investigation of how districts and charter management organizations are implementing new teacher and principal evaluation and compensation reforms and an evaluation of personalized-learning initiatives. She serves on several state and national panels on topics related to assessment, accountability, educator evaluation, and data use. She recently served as a member of the committee that revised the Standards for Educational and Psychological Testing and as chair of a What Works Clearinghouse panel on data-driven decision making. She is also an editor of Educational Evaluation and Policy Analysis. She holds an M.S. in statistics and a Ph.D. in educational psychology from Stanford University.

Brian W. Junker

Brian Junker is a professor in the Department of Statistics at Carnegie Mellon University. He is interested in highly multivariate data for which we can discover an interesting and interpretable dependence structure. He is especially intrigued when ideas in seemingly-unrelated fields come together, and so he has also been interested in capture-recapture models for estimating the size of wildlife and human populations, which share many features with models for multiple-choice tests. He has studied latent variable models employed in the design and analysis of standardized tests such as the Scholastic Aptitude Test and the Graduate Records Examination, in the analysis of small-scale experiments in psychology and psychiatry, and in the analysis of large scale educational surveys such as the National Assessment of Educational Progress. Some of his recent work aims to characterize the dependence structure implied by these models, so that one can quickly decide whether they are the right tool for a particular problem. Previously he was on the NRC Committee on Research in Education, and the Committee on Embedding Common Test Items in State and District Assessments. He served on the panel that produced NAEP: Looking Ahead Leading Assessment into the Future. He holds a B.A. in mathematics from the University of Minnesota. He also holds a M.S. in mathematics, and a Ph.D. in statistics both from the University of Illinois.

Suzanne Lane

Suzanne Lane is a professor in the research methodology program at the University of Pittsburgh. Her research and professional interests are in educational measurement and testing, with a focus on technical and validity issues in large scale assessment programs and the effectiveness of education and accountability programs. She was the president of the National Council on Measurement in Education (NCME), vice president of Division D of American Educational Research Association (AERA), member of the AERA, American Psychological Association, member of the NCME Joint Committee for the Revision of the Standards for Educational and Psychological Testing, and member of the Management Committee for the next revision of the Standards (2006-2015). She has served on the editorial boards for the Journal of Educational Measurement, Applied Measurement in Education, Educational Assessment, Educational Researcher, and Educational Measurement: Issues and Practice. She has also served on a number of technical advisory committees for the College Board, ETS, PARCC, U.S. Department of Education’s Evaluation of NAEP, U.S. Department of Education Race to the Top Technical Review Panel, NCEO, as well as for several state assessment and accountability. Previously she was a member of the NRC Committee on Test Design for K-12 Science Achievement. She holds a Ph.D. in educational psychology from the University of Arizona.

Sharon J. Lewis

Sharon J. Lewis recently retired from the position of director of research for the Council of the Great City Schools in Washington, D.C. While there she directed the council’s research program, which contributes to the organization’s efforts to improve teaching and learning in the nation’s urban schools as well as help develop education policy. She has previously worked as a national education consultant. Earlier, she was assistant superintendent of research, development and coordination, with the Detroit Public Schools. She has extensive experience with the NAS, and is currently a member of the Committee for the Five Year (2009-2013) Summative Evaluation of the District of Columbia Public Schools, and a member of the Board on Testing and Assessment. Lewis earned an M.A. in educational research from Wayne State University.

Bernard L. Madison

Bernard L. Madison is a professor in the Department of Mathematical Sciences at the University of Arkansas. Dr. Madison began his career at Louisiana State University, where for 13 years he was active in publishing mathematics research and teaching. In 1979, he accepted a position as professor and chair of Mathematical Sciences at the University of Arkansas. Ten years later, he was appointed dean of the Fulbright College of Arts and Sciences at Arkansas, a position he held until 1999. Since then he has been teaching and working on various projects in articulation, assessment, quantitative literacy and teacher education. During 2001, he was visiting mathematician at the Mathematical Association of America in Washington, DC. From 1985 to 1990, he structured and directed the program "Mathematical Sciences in the Year 2000" at the National Research Council, including the national colloquium, Calculus for a New Century. He is currently serving on the NRC’s U.S. National Commission on Mathematics Instruction. He worked with the College Board's® AP Calculus Program in several roles from 1983 to the present, including five years on the AP Calculus Committee and chief reader from 1995 to 1999. During 1999 and 2000, he served on the National Commission on the Future of the AP Program, which produced the 2001 report Access to Excellence. He currently serves as a member of the College Board's Mathematics Academic Advisory Committee and the CB Southwest Region's AP Advisory Committee. Dr. Madison holds a bachelor's degree in mathematics and physics from Western Kentucky University. He also has his master's and doctoral degrees in mathematics from the University of Kentucky.

Scott Norton

Scott Norton is strategic initiative director of standards, assessment, and accountability at the Council of Chief State School Officers. In this role, he works with states to implement the Common Core State Standards and assessments and to create and implement new student-focused accountability systems. The team is also responsible for the State Collaboratives on Assessment and Student Standards, the National Conference on Student Assessment, and collaborative work with the assessment consortia. Dr. Norton previously served as the assistant superintendent of the Office of Standards, Assessments, and Accountability at the Louisiana Department of Education. His responsibilities included the implementation of content standards and development of the Louisiana Comprehensive Curriculum as well as the state's transition to full implementation of the Common Core State Standards. Dr. Norton holds a Ph.D. in educational administration and supervision from Louisiana State University.

Sharon Vaughn

Sharon Vaughn is the H.E. Hartfelder/Southland Corp. Regents chair at the University of Texas at Austin, where she has been a professor since 1997. She is also the executive director of The Meadows Center for Preventing Educational Risk, an organized research unit at her university. Prior to this, she was a professor at the University of Miami and an assistant professor at the University of New Hampshire. Dr. Vaughn began her professional career as a classroom teacher in public schools in Arizona and Missouri. Dr. Vaughn’s research focuses on strategies and educational interventions in teaching reading to students who are at risk, particularly students with learning difficulties and behavior problems and students who are English language learners. Her work spans from the middle grades to the secondary grades. Dr. Vaughn served on the NAS Committee on Minority Representation in Special Education. She is the recipient of numerous awards including the Council for Exceptional Children research award and The University of Texas Distinguished Faculty Award and Outstanding Researcher Award. She holds a B.S. in education from the University of Missouri. She also holds a M.Ed. in education and Ph.D. in education and child development from the University of Arizona.

Lauress L. Wise

Lauress Wise is a principal scientist at the Human Resources Research Organization (HumRRO). He was president of HumRRO from 1994-2007, and stepped down in 2008 to focus more fully on research activities. Dr. Wise has over 35 years’ experience in educational research and continues extensive work on educational policy and assessment issues. He serves on technical advisory committees for the Hawaii, Oklahoma, and Rhode Island departments of education, and is currently directing the independent evaluation of California’s new high school exit exam. He recently served as co-chair for the committee that revised the AERA/APA/NCME Standards for Educational and Psychological Testing and is currently serving as president of the National Council of Measurement in Education. Dr. Wise has extensive experience with NAEP. He served on the National Academy of Education's Panel for the Evaluation of the NAEP Trial State Assessment, served on the NAS Committee on the Evaluation of NAEP, and currently directs HumRRO project to provide quality assurance for NAEP. Dr. Wise also has extensive experience with the NAS. He has served as chair of the Board on Testing and Assessment. He was also a co-principal investigator on the first year of the Congressionally-mandated evaluation of President Clinton’s Voluntary National Tests, chaired the NAS committee that performed the second year of that evaluation and chaired the Committee on Research in Education. Dr. Wise has a Ph.D. in mathematical psychology from the University of California, Berkeley.

Closed

Evaluation of National Assessment of Educational Progress Achievement Levels 2nd Meeting

May 27 - 28, 2015

Past

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Peg Goertz
Bernard Madison
Suzanne Lane
Karla Egan
Laura Hamilton
Peter Afflerbach
Sybilla Beckmann
Brian Junker
Sharon Lewis
Scott Norton

The following topics were discussed in the closed sessions:

Discuss the Open Session
Discuss Conclusions and Recommendations
Discuss the Report Plan
Discuss the Virtual Meetings

Date of posting of Closed Session Summary:

October 06, 2015

Contact

Kelly Arrington

202-334-3776
Karrington@nas.edu

Location

National Academy of Sciences Building

2101 Constitution Ave NW

Washington DC 20418, USA

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Log In

Sign in to access your saved publications, downloads, and email preferences.

Former MyNAP users: You'll need to reset your password on your first login to MyAcademies. Click "Forgot password" below to receive a reset link via email. Having trouble? Visit our FAQ page to contact support.

Members of the National Academy of Sciences, National Academy of Engineering, or National Academy of Medicine should log in through their respective Academy portals.

Email (required)
Password (required)

Don't have an account?

Register

Email (required)
New password (required)
Retype password (required) Password must be at least 8 characters
I accept the terms of use
terms of use

Thank You

Thank you for creating a MyAcademies account!

Enjoy free access to thousands of National Academies' publications, a 10% discount off every purchase, and build your personal library.

Forgot Password

Enter the email address for your MyAcademies (formerly MyNAP) account to receive password reset instructions.

Reset Requested

We sent password reset instructions to your email . Follow the link in that email to create a new password. Didn't receive it? Check your spam folder or contact us for assistance.

Reset Password

New password (required)
Retype password (required) Password must be at least 8 characters

Reset Password

Your password has been reset.

Log In

Verify email

Verify Your Email Address

We sent a verification link to your email. Please check your inbox (and spam folder) and follow the link to verify your email address. If you did not receive the email, you can request a new verification link below

Subscribe

First Name
Last Name
Email (required)
I agree to receive emails from the National Academies (required)

We respect your privacy and will never share your information.

Evaluation of National Assessment of Educational Progress Achievement Levels

Featured publication

Description

Contributors

Committee

Sponsors

Staff

Major units and sub-units

Contact

Provide feedback

Request materials

More like this

Review of the draft United by Nature assessment

Modernizing Mathematics Education for Grades 9-14

Health Care Financing to Improve Availability of Affordable Contraception: Identifying Promising Practices

Issue Paper on Mid-Scale Manufacturing for Defense

Review of the draft United by Nature assessment

Modernizing Mathematics Education for Grades 9-14

Future Directions for Social and Behavioral Science Methodologies in the Next Decade: A Workshop

Approaches to Improving the Measurement of Death by Suicide of Law Enforcement Officers in the United States: A Workshop

Ensuring the Quality, Credibility, and Relevance of U.S. Homeland Security Statistics

Publication information

Suggested citation

Additional Information

Contact

Kelly Arrington

Location

Disclaimer

My Academies

Evaluation of National Assessment of Educational Progress Achievement Levels

Dropdown items

Featured publication

Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress

Description

Contributors

Committee

Sponsors

Staff

Major units and sub-units

Contact

Provide feedback

Request materials

More like this

Review of the draft United by Nature assessment

Modernizing Mathematics Education for Grades 9-14

Health Care Financing to Improve Availability of Affordable Contraception: Identifying Promising Practices

Issue Paper on Mid-Scale Manufacturing for Defense

Review of the draft United by Nature assessment

Modernizing Mathematics Education for Grades 9-14

Future Directions for Social and Behavioral Science Methodologies in the Next Decade: A Workshop

Approaches to Improving the Measurement of Death by Suicide of Law Enforcement Officers in the United States: A Workshop

Ensuring the Quality, Credibility, and Relevance of U.S. Homeland Security Statistics