Measuring Literacy: Performance Levels for Adults (2005)

Chapter: Appendix A The Committee's Public Forums on Performance Levels for NAAL

Previous Chapter: References
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

Appendix A
The Committee’s Public Forums on Performance Levels for NAAL

Public Forum—February 27, 2004

Panelists

Cynthia Baur, U.S. Department of Health and Human Services

Beth Beuhlmann, U.S. Chamber of Commerce, Workforce Preparation

Richard Colvin, Hechinger Institute

Leslie Farr, Ohio State University

Milton Goldberg, Education Commission of the States (ECS)

Richard Long, International Reading Association

Christopher Mazzeo, National Governors Association

Gemma Santos, Miami Dade Public Schools

Tony Sarmiento, Senior Service America, Inc.

Linda Taylor, Comprehensive Adult Student Assessment System

Robert Wedgeworth, Proliteracy Worldwide

Participants

Joan Auchter, GED Testing Service/American Council on Education

Justin Baer, American Institutes for Research (AIR)

Amy Baide, Department of Homeland Security

Sandra Baxter, The National Institute for Literacy (NIFL)

Jaleh Behroozi, The National Institute for Literacy (NIFL)

Martha Berlin, Westat

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

Peggy Carr, Department of Education, National Center for Education Statistics (NCES)

June Crawford, The National Institute for Literacy (NIFL)

Elizabeth Greenberg, American Institutes for Research (AIR)

Ricardo Hernandez, Office of Vocational and Adult Education (OVAE)

Shannon Holmes, U.S. Conference of Mayors

Eugene Johnson, American Institutes for Research (AIR)

Linda Johnston Lloyd, Health Resources & Services Administration (HRSA)

Michael Jones, Office of Vocational and Adult Education (OVAE)

Cheryl Keenan, Office of Vocational and Adult Education (OVAE)

Irwin Kirsch, Educational Testing Service (ETS)

Andy Kolstad, National Center for Education Statistics (NCES)

Mark Kutner, American Institutes for Research (AIR)

Mariann Lemke, National Center for Education Statistics (NCES)

Anne Lewis, freelance journalist

Lennox McLendon, National Adult Education Professional Development Consortium

Wendy Mettger, Mettger Communications

Leyla Mohadjer, Westat

Gerri Ratliff, Department of Homeland Security

Lyn Schaefer, GED Testing Service

Peggy Seufert, American Institutes for Research (AIR)

Sondra Stein, The National Institute for Literacy (NIFL)

Lynn Thai, Department of Homeland Security

Peter Waite, Proliteracy America

Dan Wagner, National Center on Adult Literacy

Maria White, Department of Health and Human Services

Sheida White, National Center for Education Statistics (NCES)

Kentaro Yamamoto, Educational Testing Service (ETS)

Representatives from State Departments of Adult Education—April 22-23, 2004

Bob Bickerton, Massachusetts

Steve Coffman, Missouri

Donna Cornelius, Massachusetts

Cheryl King, Kentucky

Tom Orvino, New York

Ann Serino, Massachusetts

Reecie Stagnolia, Kentucky

Linda Young, Oklahoma

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

Stakeholder Questions and Alternative Versions of Performance-Level Descriptions for the National Assessment of Adult Literacy

  1. In what ways did you use the results from the 1992 NALS? What were the strengths and weaknesses of these performance levels? To what extent did these performance levels provide you with the information that you needed?

  2. NAAL measures skills in the areas of prose, document, and quantitative literacy. To what extent is it useful and informative to have different performance level descriptions for each area? Are results from the three areas of literacy used differently? If so, how?

  3. The attachment presents three alternative versions of performance-level descriptions for the prose literacy scale. Sample 1 is simply a reformatted version of the existing performance-level descriptions with 5 levels. Sample 2 is a 4-level model, and Sample 3 is a 3-level model. Please comment on how many levels are needed. What types of decisions are made at the various levels? What are the critical distinctions that need to be made?

  4. Level Labels: The three samples present different labels for the levels. Sample 1 uses numbers (Col. 2). Samples 2 and 3 use phrases as labels (Col. 2). In addition, Sample 3 presents a narrative description of the label (Col. 3). Please comment on these alternative labels. What types of labels are useful and informative? Feel free to make suggestions for alternative labels.

  5. Level Descriptions: The three samples present different ways of describing the skills represented by the performance level. Sample 1 describes the tasks associated with the level (Col. 3). Sample 2 describes what an average respondent who scores at this level should be able to do (Col. 3). Sample 3 (Col. 4) describes what the average respondent who scores at this level is able to do and not able to do in probabilistic terms (i.e., likely, not likely). Please comment on these alternative ways of describing the skills associated with the levels. What types of descriptions are useful and informative? Feel free to make suggestions for alternative descriptions.

  6. Sample Tasks: The three samples present different ways of exemplifying the tasks respondents who score at the level should be able to do. Samples 1 and 2 (Col. 4) are similar and provide examples drawn from actual assessment. Sample 3 (Col. 5) attempts to generalize from assessment tasks to real world tasks. Please comment on the extent to which these exemplifications are useful and informative.

  7. Relationships Between Prose Scores and Background Data: Samples 2 and 3 present the relationships between NAAL scores and key real-world factors as measured on the background questionnaire. Sample 2 (Col. 5) uses societal factors (income, education, voting) and Sample 3 (Col. 6) uses reading related factors. (Please be aware that the percentages in

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

cluded in these examples are purely hypothetical. If we were to recommend this format, the percentages would be based on analyses with NAAL data.) Please comment on the utility of this type of information.

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

SAMPLES ARE ON THE FOLLOWING PAGES

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

Sample 1: Five-Level Model, Based on Current PLDs for Prose Literacy

(Col. 1)

Level

(Col. 2)

Label

(Col. 3)

Description of tasks

(Col. 4)

Sample NAAL tasks associated with the level

I

Level 1

Most of the tasks in this level require the reader to read relatively short text to locate a single piece of information which is identical to or synonymous with the information given in the question or directive. If plausible but incorrect information is present in the text, it tends not to be located near the correct information.

  • Locate one piece of information in a sports article.

  • Identify the country in a reading passage.

  • Underline sentence explaining action stated in short article.

II

Level 2

Some tasks in this level require readers to locate a single piece of information in the text; however, several distractors or plausible but incorrect pieces of information may be present, or low-level inference may be required. Other tasks require the reader to integrate two or more pieces of information or to compare and contrast easily identifiable information based on a criterion provided in the question or directive.

  • Underline the meaning of a term given in government brochure on supplemental security income.

  • Locate two features of information in sports article.

  • Interpret instructions from an appliance warranty.

III

Level 3

Tasks in this level tend to require readers to make literal or synonymous matches between the text and information given in the task, or to make matches that require low-level inferences. Other tasks ask readers to integrate information from dense or lengthy text that contains no organizational aids

  • Write a brief letter to complain about an error on a bill.

  • Read a news article and identify a sentence that provides interpretation of a situation.

  • Read lengthy article to identify two behaviors that meet a stated condition.

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

 

 

such as headings. Readers may also be asked to generate a response based on information that can be easily identified in the text. Distracting information is present, but is not located near the correct information.

 

IV

Level 4

These tasks require readers to perform multiple-feature matches and to integrate or synthesize information from complex or lengthy passages. More complex inferences are needed to perform successfully. Conditional information is frequently present in tasks in this level and must be taken into consideration by the reader.

  • State in writing an argument made in a newspaper article.

  • Explain differences between two types of employee benefits.

  • Contrast views expressed in two editorials on technologies available to make fuel-efficient cars.

V

Level 5

Some tasks in this level require the reader to search for information in dense text which contains a number of plausible distractors. Others ask readers to make high-level inferences or use specialized background knowledge. Some tasks ask readers to contrast complex information.

  • Compare approaches stated in a narrative on growing up.

  • Summarize two ways lawyers may challenge prospective jurors.

  • Interpret a phrase from a lengthy news article.

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

Sample 2: Four-Level Model for Prose Literacy

(Col. 1)

Level

(Col. 2)

Label

(Col. 3)

The average respondent who scores at this level should be able to:

(Col. 4)

Sample of NAAL tasks the average respondent should be able to:

(Col. 5)

Relationships with societal factors

I

Below Basic

  • Identify letters and numbers.

  • Point to orally presented words.

  • Read aloud words, phrases. or short sentences.

  • Locate specific information on a food label.

  • Identify a specific word in an advertisement.

  • Locate an address in an advertisement.

  • >50% of being poor or near poor

  • >50% chance of not having a U.S. high school diploma

  • <50% chance of voting in most recent election

II

Basic

  • Locate a single piece of information in a brief written text that uses organizational aids such as headings.

  • Make simple inferences.

  • Provide appropriate answers to questions that require integrating two or more pieces of information.

  • Locate one piece of information in a sports article.

  • Identify the country in a reading passage.

  • Interpret brief instructions from warranty information.

  • 25-50% of being poor or near poor

  • 50% chance of having at least a high school diploma

  • 50-60% chance of voting in most recent election

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

III

Intermediate

  • Generate a response based on information that is easily identifiable in dense or lengthy text.

  • Integrate information from dense or lengthy text.

  • Write a brief letter to complain about an error on a bill.

  • Read a magazine article and identify the facts that support a specific inference.

  • 10-25% of being poor or near poor

  • 50% chance of having at least some college

  • 60-70% chance of voting in recent election

IV

Advanced

  • Search for information that is in dense texts.

  • Integrate or synthesize information from complex texts.

  • Perform complex inferences.

  • State in writing an argument made in a newspaper article.

  • Explain the differences between two types of employee benefits.

  • Contrast views expressed in two editorials.

  • <10% of being poor or near poor

  • 50% chance of having at least an associates college degree

  • 70-80% chance of voting in recent election

NOTES: NAAL background data will be analyzed to provide information about the likely relationships between literacy levels and the three societal factors included in Column 5; the current figures are for demonstration purposes only. Many of these descriptions may not generalize to English language learners (ELL). ELLs may be literate in languages other than English, and the relationships with societal factors (Col. 5) may be different for ELLs than for native English speakers. The response mode may also affect ELLs’ performance, since writing skills in English may develop at a slower pace than reading or speaking skills.

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

Sample 3: Three-Level Model for Prose Literacy

(Col. 1)

Level

(Col. 2)

Label

(Col. 3)

Description of label for individuals who score at this level:

(Col. 4)

The average respondent who scores at this level:

(Col. 5)

Sample real-world tasks the average respondent at this level should be able to do:

(Col. 6)

Relationship to reading-oriented factors

I

Minimally Literate in English

Are not able to independently handle most of the tasks of daily living that require literacy skills in English.

  • Is likely to be able to identify letters and numbers; point to words when they are presented orally; orally read individual words, phrases, or short sentences.

  • Is not likely to able to read connected text.

  • Place signature in proper place on a form.

  • Follow a simple recipe.

  • Identify an address in an advertisement.

  • >80% chance of never reading the newspaper

  • >80% chance of needing a lot of help with printed information

II

Somewhat Literate in English

Should be able to independently handle some of the tasks of daily living that require literacy skills in English.

  • Is likely to be able to locate a single piece of information in a brief written text with organizational aids, such as headings; to integrate two or more pieces of information; and to compare and contrast easily identifiable information.

  • Follow directions on a medicine bottle.

  • Read aloud to a child in preschool or elementary grade.

  • 40-80% chance of never reading the newspaper

  • 40-80% chance of needing a lot of help with printed information

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.

 

 

 

  • Is not likely to be able to integrate or synthesize information from dense text with no organizational aids.

 

 

III

Fully Literate in English

Should be able to independently handle most of the tasks of daily living that require literacy skills in English.

  • Is likely to be able to integrate or synthesize information from dense or lengthy text that contains no organizational aids; and to generate a response based on information that can be identified in the text.

  • Read and understand New York Times.

  • Understand a proposition on a ballot.

  • <10% chance of never reading the newspaper

  • <10% chance of needing a lot of help with printed information

NOTES: NAAL background data will be analyzed to provide information about the likely relationships between literacy levels and the reading oriented factors in Column 6; the current figures are for demonstration purposes only. These descriptions may not generalize to English language learners (ELL). ELLs may be literate in languages other than English, and their performance on real world tasks (Col. 5 and 6) may be different than for native English speakers. The response mode may also affect ELLs’ performance, since writing skills in English may deve lop at a slower pace than reading or speaking skills.

Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 203
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 204
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 205
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 206
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 207
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 208
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 209
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 210
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 211
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 212
Suggested Citation: "Appendix A The Committee's Public Forums on Performance Levels for NAAL." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
Page 213
Next Chapter: Appendix B Examination of the Dimensionality of NALS
Subscribe to Email from the National Academies
Keep up with all of the activities, publications, and events by subscribing to free updates by email.