This appendix details how the bookmark procedure was implemented and reports results for the committee’s September session. Follow ing the text are the background materials, which include the agenda, participant questionnaires, tables, and figures for the September session.

A total of 30 panelists from the fields of adult education, middle and high school English language arts, industrial and organizational psychology, and state offices of adult education participated in the second standard setting, held over three days in September 2004. Six of the panelists had participated in the July standard setting. These six individuals returned in September as table leaders, which added continuity of process and familiarity of material to the second session (the agenda is included in Background Materials at the end of this appendix).

As in July, panelists were given a questionnaire to collect background information (a blank questionnaire is included in Background Materials at the end of this appendix). Almost half (46.7 percent, n = 14) of the September participants had managerial responsibilities for adult education in their states or regional areas, although several (20 percent, n = 6) were also instructors in adult education. Half (50 percent, n = 14) of the participants who completed the questionnaire reported they were somewhat familiar with NAAL prior to participating in the standard-setting activities; five (17.9 percent) reported that they were very familiar with NAAL, and nine (32.1 percent) said they were unfamiliar with NAAL prior to the standard

setting. In addition, participants responded that their work environments were predominantly urban (48.1 percent, n = 13) or suburban (37 percent, n = 10).

On the basis of the primary responsibilities listed on their resumes, the 29 panelists were classified into five areas of expertise: Adult Basic Education (ABE), General Educational Development (GED), English for Speakers of Other Languages (ESOL), middle or high school language arts (grades 6-12), and industrial and organizational psychology. Participants were randomly assigned to one of six tables of five people. Four of the six tables had a representative from each of the five areas of expertise; one table included a workplace and labor force literacy expert.

Once panelists were assigned to tables, the groups were then randomly assigned to literacy areas using the same counterbalancing design used in July (Table D-1). Two tables worked on prose literacy first; one of these tables was then assigned to work on document literacy and the other to work on quantitative literacy. Two tables worked on document literacy first; one of these tables was assigned to work on quantitative literacy and the other to work on prose literacy. The remaining two tables that worked on quantitative literacy first were similarly divided for the second content area: one table was assigned to work on prose literacy while the other was assigned to work on document literacy. Again, the bookmark placements were designated as Occasion 1 or Occasion 2 to indicate the order with which the table work on each assigned literacy area.

The ordered item booklets used for the second standard setting were organized in the same way as for the first standard setting. One small change, however, was that some of the NAAL test questions were scored according to a partial credit scheme. This means that answers were scored as wrong, partially correct, or fully correct. When a partial credit scoring scheme is used, a difficulty value is estimated for both the partially correct score and the fully correct score. As a result, the test questions have to appear multiple times in the ordered item booklet, once for the difficulty value associated with partially correct and a second time for the difficulty value associated with fully correct. The ordered item booklets included the scoring rubric for determining partial credit and full credit scores.

Training procedures in September were similar to those used in July. Table leader training was held the day before the standard setting, and panelist training was held on the first day of the standard setting. The

majority of materials presented to the September panelists during the three-hour training session were the same as those presented in July, and most of the procedures were the same.

The procedures used in September were similar to those used in July, with the exception that the committee decided that all panelists in September should use the instructions for a response probability of 67 percent. This meant that the design for the standard setting could follow more typical bookmark procedures. That is, groups of panelists usually work on the same ordered item booklet at different tables during Rounds 1 and 2 but join each other for Round 3 discussions. Therefore, in September, the two tables working on the same literacy area were merged for the Round 3 discussion.

During Round 3, panelists received data summarizing bookmark placements for the two tables combined. This included a listing of each panelist’s bookmark placements and the median bookmark placements by table. In addition, the combined median scale score (based on the data from both tables) was calculated for each level, and impact data provided about the percentages of adults who would fall into the below basic, basic, intermediate, and advanced categories if the combined median values were used as cut scores. (Because the full 2003 NAAL data set was not ready in time for the standard setting, the impact data used for Round 3 were based on the 1992 NALS results.) Panelists from both tables discussed their reasons for choosing different bookmark placements, after which each panelist independently made his or her final judgments about bookmark placements for the basic, intermediate, and advanced literacy levels.

As in July, panelists in September were asked to complete a satisfaction questionnaire about their perception of the standard-setting process at the end of the session.¹ The majority (93 percent, n = 28) reported that they were very satisfied with the organization of the event (Question 7) and that they were either satisfied (30 percent, n = 9) or very satisfied (63 percent, n = 19) with the cut score decisions of their table (Question 8).

As in July, panelists were also asked, questions about their background experiences with adult education and their familiarity with NAAL prior to the standard-setting session. Questions added to the panelist professional and personal information questionnaire (see page 299) based on feedback

from the July participants included a series of three questions on how well participants understood the context of the test, the meaning of the performance levels, and the meaning of the bookmark placement. Results from this questionnaire were positive. A total of 28 panelists (93 percent) reported that they were very comfortable with the context of the test, the meaning of the performance levels, and the meaning of the bookmark placement (the remaining two individuals did not complete this part of the questionnaire).

At the conclusion of the September standard setting, 12 of the panelists were asked to stay for an extended session to write performance-level descriptions for the NAAL items. The panelists represented a cross-section of the larger group, in that at least one member from each of the six tables particpated in the extended session and there was representation as well from each of the three areas of expertise (adult education, middle and high school English language arts, and industrial and organizational psychology). The 12 participants were split into 3 groups of 4, with each group focusing on one of the three NAAL literacy areas. A period of approximately two hours was allotted for the panelists to discuss and suggest revisions to the performance-level descriptions. At this point, specific examples of, and references to, items and stimuli in the released NALS items were incorporated into the performance-level descriptions.²

The methods for the September standard setting were, for the most part, the same as those used during the July session with respect to the sequencing of the standard-setting activities. The primary difference between the July and September sessions was that the committee had decided, partly on the basis of the analyses of the July standard-setting data, that panelists would use only a response probability of 67 percent (rp67) in the September session. This decision allowed the standard-setting design to be fully counterbalanced (Table D-1). That is, panelists at Tables 1 and 2 worked with prose items during Occasion 1; during Occasion 2, panelists at Table 1 worked with the document literacy items while Table 2 panelists worked with the quantitative items. Another difference between the July and September sessions was that, in July, use of multiple rp assignments

precluded having panelists from different tables join each other during Round 3. Because all panelists in September used the same rp level, those working on the same literacy area at two different tables were able to merge into one table for the Round 3 discussion, a practice advocated by the developers of the bookmark procedure (Mitzel et al., 2001). Results from this session are reported below.

A complete listing of all judgments made by each panelist who reviewed the prose literacy scale at the September standard-setting session is presented in Tables D-2A, D-2B, and D-2C. The information included in the table consists of each participant’s bookmark placement for each round, as well as the corresponding scale score.³ The table number used by each panelist is provided, as well as an indication of whether a given literacy scale was reviewed by the panelist first (i.e., Occasion 1) or second (i.e., Occasion 2).

Figure D-1 provides a visual depiction of the cut scores associated with panelists’ bookmark placement decisions across the three rounds. These graphs are presented in a slightly different manner than for the July standard setting. Tables 1 and 2 examined the prose ordered item booklets in Occasion 1 (top two graphs), and unlike the July standard setting, panelists at these two tables joined each other after Round 2. Therefore, the graphs are presented as mirror opposites—the top left-hand graph shows the placements moving from Round 1 to Round 3; the top right-hand graph shows the placements moving from Round 3 to Round 1. This provides a means for easily comparing the extent of agreement across the two tables after Round 3. The bottom two graphs show the same information for Tables 4 and 5 working with prose items during Occasion 2.

Overall, the variability in panelists’ cut scores tended to decrease across the rounds, particularly for the basic and intermediate performance levels. At Tables 1 and 2, considerable variability was evident in the advanced level cut scores, even at Round 3; agreement about the advanced level cut scores was better for Tables 4 and 5.

A summary of the Round 3 combined (Occasion 1 and 2) cut scores for prose literacy from July and September is given in Table D-3. The variability in the advanced cut score is evident in the standard deviations in this table.

A complete listing of all judgments made by each panelist who reviewed the document literacy scale at the September standard setting session is presented in Tables D-4.

Figure D-2 portrays the cut scores associated with panelists’ bookmark placements for each of the three rounds. Here, Tables 3 and 4 reviewed the document items during Occasion 1 (top two graphs); and Tables 1 and 6 reviewed the document items during Occasion 2 (bottom two graphs). Again, convergence in the cut scores is apparent by Round 3 for the basic and intermediate performance levels. Considerable disparity is present for the advanced level at all tables except Table 4.

A summary of the Round 3 combined (Occasion 1, Occasion 2) cut scores for document literacy for July and September is given in Table D-5. Again, the divergence in opinion about the advanced cut scores is evident in the size of the standard deviation.

A complete listing of all judgments made by each panelist who reviewed the quantitative literacy scale at the September standard setting session is presented in Tables D-6A, D-6B, D-6C. Figure D-3 presents the cut scores associated with panelists’ bookmark decisions across the three rounds. Table 5 and 6 reviewed the quantitative literacy items during Occasion 1 (top two graphs); Tables 2 and 3 reviewed the items during Occasion 2 (bottom two graphs). Overall, there was a trend toward consensus by Round 3, although there was notable variability in cut scores for the advanced level at Table 5 and the basic level at Table 6, even at Round 3. Panelists verbally reported after the standard-setting session that they had more difficulty placing bookmarks for the quantitative section than they did for the prose or document section. A summary of the Round 3 combined (Occasion 1, Occasion 2) cut scores for document literacy from the July and September sessions is given in Table D-7.

Although the bookmark method does not require panelists to reach consensus on the cut scores, agreement is encouraged. One indication of the quality of the standard-setting process is an examination of the extent to which they agreed with regard to their bookmark placements. This level of agreement is evaluated through estimates of interrater agreement.

To calculate the interrater agreement, we used the reliability feature in Statistical Package for the Social Sciences (SPSS) to estimate the intraclass

correlations among the Round 3 scaled cut scores for each literacy area. These intraclass correlations appear below.

These values are all quite high and indicate that rater agreement was at acceptable levels.

During the course of our analytic work on the 2003 NAAL data, we received several versions of the data files from NCES and its contractor. The timing of the receipt of these files did not always coincide with the schedule for the committee’s work, and this may necessitate that NCES and its contractors repeat some of the committee’s analyses. To facilitate replication, should it be necessary, we provide additional details from the bookmark standard setting and specify which data files we used for the different stages of our work.

Tables D-8 through D-13 provide additional information from the bookmark standard setting. Each table gives the Round 3 bookmark placements and corresponding cut score by participant, table, response probability value, and occasion for each of the six standard-setting groups (prose, document, and quantitative literacy from July and September sessions). Also included is the identification number of each item on which a bookmark was placed. On these tables, the cut score is the scale score corresponding to the proficiency estimate (given the specified response probability criterion) for the item just before the bookmark placement.

The item parameters used for the July bookmark standard setting were those in the publicly available data file. The transformation constants used to convert IRT proficiency estimates to scaled scores appeared in footnote 3 of Appendix C (p. 228).

The item parameters used for the September bookmark standard setting were those on the file forwarded to us in August 2004, which was based on data for the main NAAL sample but did not include the additional state and inmate samples. The transformation constants used to convert IRT proficiency estimates to scaled scores appeared in footnote 3 of Appendix D (p. 289).

All of the analyses of 2003 test takers (e.g., the population percentages at each performance level, the median literacy scores derived for the quasi-contrasting group procedure) were based on the file delivered to us on January 21, 2005. The transformation constants (scale and location) used to convert IRT proficiency estimates to scaled scores were for prose, 58.480557 and 280.704956; for document, 58.755463 and 274.881560; and for quantitative, 63.311586 and 280.488425.

The committee wishes to acknowledge the assistance and contributions of the individuals who served as panelists for the two bookmark standard settings and provided valuable input on the performance-level descriptions.

Eunice Askov, Pennsylvania State University

Marjorie Ball, Mississippi State Penitentiary, Parchman

Roxanne Bauer, Indianapolis Public Schools, Indiana

Michelle Blantz, South Georgia Technical College

Rhodella Brown, Daytona Beach Community College, Florida

Miriam Burt, Center for Applied Linguistics, Washington, DC

Laura Chenven, AFL-CIO Working for America Institute, Washington, DC

Suzanne Cimochowski, EASTCONN, Hampton, Connecticut

Marie Cora, Hotspur Partners, LLC, Boston

Christopher Coro, Northampton Community College, Pennsylvania

Susan Cowles, Oregon State Department of Community Colleges and Workforce Development

Shari Crockett, Regional Office of Education, Monroe/Randolph Counties, Illinois

Lansing Davis, New Jersey State Employment and Training Commission

Kim Donehower, University of North Dakota

Suzanne Elston, Bradley County Adult Education, Tennessee

Leslie Farr, Ohio State University

Sharon Floyd, Saginaw Public Schools, Michigan

Janet Geary, North Kansas City School District, Missouri

Karen Gianninoto, Salisbury State University, Maryland

Kimberly Gibson, Sierra College, California

Suzanne Grant, Arlington Public Schools, Virginia

Anne Greenwell, Jefferson County Public Schools, Kentucky

Christina Gutierrez, T.C. Williams High School, Alexandria, Virginia

Nancy Hampson, San Diego Community College District, California

James Harris, Caliber Associates, Fairfax, Virginia

Roberta Hawkins, Shorewood High School, Shoreline, Washington

Fran Holthaus, Upper Valley Joint Vocational School, Piqua, Ohio

Sally House, Central Mississippi Correctional Facility, Pearl

Brenda Jeans, Beauregard Parish School Board, Louisiana

Paul Jurmo, New York University

Judy Kihslinger, Waukesha County Technical College, Wisconsin

Terry Kinzel, Big Bend Community College, Washington

Jaqueline Korengel, Commonwealth of Kentucky, Frankfort

Nathan Kuncel, University of Illinois at Urbana-Champaign

Diane Lindahl, Western Wisconsin Technical College

Ardith Loustalet, St. Vrain Valley School District, Colorado

Alfredo Lujan, Monte del Sol Charter School, New Mexico

Sanford Marks, Community College of Southern Nevada

Peggy McGuire, University of Tennessee

Maureen Meehan, University of Illinois at Chicago

Doug Molitor, 3M, St. Paul, Minnesota

Donald Mott, Wilson Mott & Associates, Greenville, North Carolina

Vivian Mott, East Carolina University

Bill Muth, U.S. Federal Bureau of Prisons, Washington, DC

Connie Nelson, Massachusetts Worker Education Roundtable, Boston

Donna Nola-Ganey, Louisiana Department of Education, Baton Rouge

Peg Perri, Western Wisconsin Technical College

Rebecca Rogers, Washington University in St. Louis

Teresa Russell, Independent Consultant, Minnesota

Sally Sandy, Parkway School District, Missouri

Kathleen Santopietro Weddel, Colorado Department of Education

Diane Schroeder, St. Charles Community College, Missouri

Don Seaman, Texas Center for the Advancement of Literacy and Learning, College Station

Jane Siveria, Florida Department of Education, Tallahassee

Cristine Smith, World Education, Inc., Boston

Maggie Sokolik, University of California, Berkeley

Linda Stacy, Owens Community College, Ohio

Linda Taylor, Comprehensive Adult Student Assessment System, San Diego

Ray Thompson, Middle Georgia Technical College

Patricia Thorpe, University of Phoenix

Fran Tracy-Mumford, Delaware Department of Education, Dover

Karen Valbrun, Georgia State Department of Technical and Adult Education

Denise Weiner, Delaware Department of Education, Dover

Lynne Weintraub, Jones Library, Amherst, Massachusetts

Ira Yankwitt, Literacy Assistance Center, New York

Linda Young, Oklahoma State Department of Education

September Standard-Setting Session

Item

Page

Standard-Setting Session for the

National Assessment of Adult Literacy (NAAL)

National Research Council, Washington, DC

September 17-20, 2004

Friday, September 17, 2004—The Day Before the Standard Setting

Saturday, September 18, 2004—Day 1 of Standard Setting

Sunday, September 19, 2004—Day 2 of Standard Setting

Monday, September 20, 2004—Day 3 of Standard Setting

Bookmark Standard-Setting Session for NAAL

September 18-20, 2004

National Research Council, Washington, DC

Please answer the following questions so we better understand the characteristics of our group of standard setting participants.

1. Do your professional responsibilities include direct or managerial responsibilities ‘for the education of adults? _____________________________

___ Yes. I am directly involved as an instructor. For how many years have you had such responsibilities? __________________________________

___Yes. I am involved in a managerial capacity. For how many years have you had such responsibilities? __________________________________

___Yes. I am directly involved with the education of adults but not as an instructor or a manger. In what way are you involved with the education of adults? For how many years have you had such responsibilities? _______

___No. Please characterize your professional responsibilities:

_____________________________________________________________

2. How would you characterize the educational setting for these adults (check any and all that apply):

3. How familiar were you with the National Assessment of Adult Literacy (a.k.a. NAAL) before your participation in the standard-setting activities?

_____ Unfamiliar _____Somewhat familiar ____Very familiar

4. Standard-setting judgments require one to understand: (A) the content of the test (i.e., literacy), (B) the performance-level descriptions, and (C) the standard-setting task (i.e., what it means to place a bookmark in an ordered item book).

A. How well did you understand the content of the test? Please circle one number for each area of literacy for which you set a standard:

B. How well did you understand the meaning of the performance levels as explained in the performance-level descriptions?

C. How well did you understand the meaning of bookmark placement based on your training and the bookmark placement instructions?

5. Please tell us about yourself (optional)

Gender: ____Male ____Female

Age: ____20-29 ____30-39 ____40-49 ____50-59 ____60-69 ____70+

Race/Ethnicity: _______________________

Type of community in which you work: