As described in the body of the report, the Committee on Performance Levels for Adult Literacy convened two bookmark standard-setting sessions in 2004, one in July to gather panelists’ judgments about cut scores for the 1992 National Adult Literacy Survey (NALS) and another in September to collect judgments about cut scores for the 2003 National Assessment of Adult Literacy (NAAL). This appendix details how the bookmark procedure was implemented and reports results for the July session, and Appendix D presents similar information for the September session. Following the text are the background materials, which include the agenda, participant questionnaires, tables, and figures for the July session. The appendix concludes with technical details about the data files that the committee used for the standard settings; this information is provided to assist the U.S. Department of Education and its contractors with any follow-up analyses that need to be conducted with respect to the cut scores for the performance levels.

The July 2004 session was held to obtain panelists’ judgments about cut scores for the 1992 NALS and to collect their feedback about the performance-level descriptions. Several consultants assisted the committee with the standard setting, including Richard Patz, one of the original developers of the bookmark procedure.

A total of 42 panelists participated in the standard setting. Background information on the panelists was collected by means of a questionnaire (a

blank questionnaire is included in Background Materials at the end of this appendix). A majority (85 percent, n = 28) had managerial responsibilities for adult education in their states or regional areas, although many panelists were instructors as well as program coordinators or directors. Most panelists worked in adult basic education (66 percent, n = 22), general educational development or GED (54 percent, n = 18), or English language instruction (51 percent, n = 17) settings. Almost half (45 percent, n = 15) reported they were very familiar with NALS prior to participating in the standard-setting activities; 42 percent (n = 14) reported that they were somewhat familiar with NALS. Only four participants (12 percent) who completed the questionnaire said they were unfamiliar with NALS prior to the standard setting.

Panelists were assigned to tables using a quasi-stratified-random procedure intended to produce groups with comparable mixtures of perspectives and experience. To accomplish this, panelists were assigned to one of nine tables after being sorted on the following criteria: (1) their primary professional responsibilities (instructor, coordinator or director, researcher), (2) the primary population of adults they worked with as indicated on their resumes, and (3) the areas in which they worked as indicated on their resumes. The sorting revealed that panelists brought the following perspectives to the standard-setting exercise: adult basic education (ABE) instructor, English for speakers of other languages (ESOL) instructor, GED instructor, program coordinator or director, or researcher. Panelists in each classification were then randomly assigned to one of the nine tables so that each group included at least one person from each of the classifications. Each table consisted of four or five panelists and had a mixture of perspectives: instructor, director, researcher, ESOL, GED, and ABE.

Once panelists were assigned to tables, each table was then randomly assigned to two of the three literacy areas (prose, document, or quantitative). The sequence in which they worked on the different literacy scales was alternated in an attempt to balance any potential order effects (see Table C-1). Three tables worked with the prose items first (referred to as Occasion 1 bookmark placements) and the document items second (referred to as Occasion 2 bookmark placements); three tables worked with the document items first (Occasion 1) and the quantitative items second (Occasion 2); and three tables worked with the quantitative items first (Occasion 1) and the prose items second (Occasion 2).

For each literacy area, an ordered item booklet was prepared that rank-ordered the test questions from least to most difficult according to the responses of NALS examinees. The ordered item booklets consisted of all

the available NALS tasks for a given literacy area, even though with the balanced incomplete block spiraling design used for the assessment, no individual actually responded to all test questions. The tasks were arranged in the ordered item booklets so that the question appeared first (one question per page) followed by the stimulus materials (e.g., a newspaper article, a bus schedule, a graph) and the scoring rubric. Accompanying each ordered item booklet was an item map that listed each item number and a brief description of the item. The number of items in each NALS ordered item booklet was 39 for prose literacy, 71 for document literacy, and 42 for quantitative literacy.

Two training sessions were held, one just for the table leaders, the individuals assigned to be discussion facilitators for the tables of panelists, and one for all panelists. The role of the table leader was to serve as a discussion facilitator but not to dominate the discussion or to try to bring the tablemates to consensus about cut scores. Table leaders also distributed standard-setting materials to each table member, guided the discussions of the content and context characteristics that differentiated NALS test items from each other, led the discussion of the impact data for the final round of bookmark placements, and ensured that security procedures were followed. Table leader training was held the day before the standard setting to familiarize the table leaders with their roles, the NALS materials, and the agenda of activities for the standard-setting weekend. (The agenda for the July session is included in Background Materials at the end of this appendix.) Panelist training was held the morning of the standard setting. Richard Patz facilitated both training sessions and used the same training materials for both sessions. This helped ensure that the table leaders were well acquainted with the bookmark process.

The training began with an overview of NALS (skills assessed by the tasks in the three literacy areas, administrative procedures, etc.), followed by background about the committee’s charge and the timing of its work. Panelists were told that the cut scores that resulted from the bookmark procedure would be the group’s recommendations to the committee but that it would ultimately be up to the committee to determine the final cut scores to recommend to the Department of Education. Panelists then received instruction in the elements and procedures of the bookmark method.

Once the training session was completed, the bookmark process began by having each panelist respond to all the questions in the NALS test

booklet for their assigned literacy scale. For this task, the test booklets contained the full complement of NALS items for each literacy scale, arranged in the order that test takers would see them but not ranked-ordered as in the ordered item booklets. Afterward, the table leader facilitated discussion of differences among items with respect to knowledge, skills, and competencies required and familiarized panelists with the scoring rubrics. Panelists were expected to take notes during the discussion, which would be used in making their judgments.

Panelists then received the ordered item booklets. They discussed each item and noted characteristics they thought made one item more difficult than another. The table leader distributed the performance-level descriptions.¹ Each table member then individually reviewed the performance-level descriptions, the items in the ordered item booklet, the scoring rubrics, and their notes about each item and proceeded to independently place bookmarks to represent cut points for basic, intermediate, and advanced literacy; this first bookmark placement constituted Round 1.

On the second day of standard setting, each table received a summary of the Round 1 bookmark placements made by each table member and were provided the medians of the bookmark placements (calculated for each table). Table leaders facilitated discussion among table members about their respective bookmark placements, moving from basic to intermediate to advanced literacy, without asking for consensus. Panelists were given just under two hours to deliberate about differences in their bookmark placements before independently making judgments for Round 2. Throughout the standard setting, staff members, consultants, assistants, and four committee members observed the interactions among the panelists as they discussed the characteristics of the items and their reasons for selecting their bookmark placements.

For Round 3, each table again received a summary of the Round 2 bookmark placements made by each table member as well as the medians for the table. In addition, each table received impact data, that is, the proportion of the 1992 population who would have been categorized at the below basic, basic, intermediate, or advanced literacy level based on the table’s median cut points. After discussion of the variability of Round 2 judgments and the impact of their proposed cut points on the percentages of adults who would be placed into each of the four literacy groups, each panelist made his or her final judgments about bookmark placements for the basic, intermediate, and advanced literacy levels. This final set of judgments concluded Round 3.

After Round 3, panelists were asked to provide feedback about the

performance-level descriptions by reviewing the items that fell between each of their bookmarks and editing the descriptions accordingly. That is, the items in the booklet up to, but not including, the first bookmark described the basic literacy level. Panelists reviewed these items and revised the descriptions to better fit the items that fell within this level. They were asked to do the same for the intermediate and advanced performance-level descriptions.

On the afternoon of the second day, the processes described above were repeated for the second literacy area. Round 1 was completed on the second day; Rounds 2 and 3 were completed on the third day. The standard setting concluded with a group session to obtain feedback from the panelists.

In conjunction with the July standard setting, the committee collected information about the impact of varying the instructions given to panelists with regard to the criteria used to judge the probability that an examinee would answer a question correctly (the response probability). The NALS results were reported in 1992 using a response probability of 80 percent, a level commonly associated with mastery tests. Some researchers have questioned the need for such a strict criterion for an assessment like NALS, for which there are no individual results, and recommend instead using a more moderate response probability level of 67 percent (e.g., Kolstad, 2001). The authors of the bookmark method also recommend a 67 percent response probability level (Mitzel et al., 2001). Because the issue of response probability had received so much attention in relation to NALS results, the committee arranged to collect data from panelists about the impact of using different (50, 67, or 80 percent) response probability values. Specifically, we were interested in evaluating (1) the extent to which panelists understand and can make sense of the concept of response probability level when making judgments about cut scores and (2) the extent to which panelists make different choices when faced with different response probability levels. Panelists were told that they would be given different instructions to use in making their judgments and that they should not discuss the instructions with each other.

As described earlier, the panelists were grouped into nine tables of four or five panelists each. Each group was given different instructions and worked with different ordered item booklets. Three tables (approximately 15 panelists) worked with booklets in which the items were ordered with a response probability of 80 percent and received instructions to use 80 percent as the likelihood that the examinee would answer an item correctly. Similarly, three tables used ordered item booklets and instructions consistent with a response probability of 67 percent, and three tables used or-

dered item booklets and instructions consistent with a response probability of 50 percent.

Panelists received training in small groups about their assigned response probability instructions. The additional training session gave detailed instructions to panelists on one of three difficulty levels (50, 67, or 80 percent). These specialized instructions are summarized in Background Materials at the end of this appendix. Each table of panelists used the same response probability level for the second content area as they did for the first.

The performance-level descriptions used at the July standard setting consisted of overall and subject-specific descriptors for the top four performance levels. In accord with typical bookmark procedures, concrete examples of stimulus materials (e.g., newspaper articles, almanac) or types of tasks (e.g., read a bus schedule, fill out an employment application form) had been intentionally omitted from the performance-level descriptions because including specific examples tends to overly influence panelists’ judgments about the bookmark placements. Omission of specific examples allows the panelists to rely on their own expertise in making judgments.

Panelists’ written comments about and edits of the performance levels were reviewed. Many panelists commented about the lack of concrete examples, saying that a few examples would have helped them. Some were concerned that NALS did not have enough items at the upper end of the spectrum for them to confidently make a distinction between intermediate and advanced categories. They also suggested edits, such as adding the modifier “consistently” to the levels higher than below basic, asked for clarification of adjectives such as “dense” versus “commonplace” text and “routine” versus “complex” arithmetic operations. In addition, the panelists raised questions about the scope of the NALS quantitative assessment and the extent to which it was intended to evaluate arithmetic skills versus functional quantitative reasoning. They also pointed out inconsistencies in the wording of the descriptions, moving from one level to the next. The committee used this feedback to rethink and reword the level descriptions in ways that better addressed the prose, document, and quantitative literacy demands suggested by the assessment items.

Revised descriptions were used for the September standard-setting session. The following types of changes were made. The introduction to the descriptions was rewritten to include the phrase, “An individual who scores at this level, independently, and in English …,” reflecting the nature of the NALS and NAAL as tests of literacy in English in which examinees complete the test items with minimal or no help from the interviewer or other

family members or individuals. In addition, the subject-area descriptions were revised to better reflect the range of literacy skills needed for the NALS items without specifying the types of NALS items or stimuli used. Four panelists who had participated in the July standard-setting session were invited to review the revised performance-level descriptions prior to the September standard setting, and their feedback was used to further refine the descriptions.²

At the end of the July standard-setting session, panelists were asked to complete a satisfaction questionnaire (a blank questionnaire is included in Background Materials at the end of this appendix). Almost all of the participants reported that they were either very satisfied (59 percent, n = 20) or satisfied (35 percent, n = 12) with the standard-setting training, while only two participants reported that they were not satisfied with the training they received. Almost three-quarters of the participants (74 percent, n = 25) reported being very satisfied with their table interactions and discussions; roughly a quarter (26 percent, n = 9) reported that they were satisfied with the logistical arrangements. The contributions and guidance of the table leaders were perceived as mainly very satisfactory (53 percent, n = 18) or satisfactory (32 percent, n = 11). Only two participants (6 percent) indicated that their table leaders were not satisfactory. Both of these individuals wrote on their evaluations that their table leaders were overly talkative and did not facilitate discussions among the table members. The majority of comments indicated that participants thought their table leaders were well organized, adept at facilitating discussion, and kept the table members focused on the standard setting tasks.

The organization of the standard-setting session was well received: over half of the participants (68 percent, n = 23) were very satisfied and 32 percent (n = 11) reported satisfaction with the session. Participants also reported being satisfied with their work during the standard setting—94 percent of the participants reported that they were either very satisfied (44 percent, n = 15) or satisfied (50 percent, n = 17) with the cut scores decided by their table, indicating a high level of participant confidence in both the process and the product of the standard-setting session. In addition, 85 percent (n = 29) and 12 percent (n = 4) reported that participation in the standard-setting session was very valuable or valuable to them, respectively.

Besides giving feedback on the standard-setting session, panelists were also very helpful in suggesting ways in which the September standard-setting session would benefit from the perspective of those who had just completed the process. For example, the participants reflected a range of adult education areas, such as ABE, GED, and ESL. While the experiences and perspectives of these individuals were useful and appropriate for the standard-setting task, the July participants asked that the committee consider broadening the array of perspectives for the September gathering by including middle school or high school language arts teachers and professionals familiar with human relations, employment testing, or skills profiling. The July participants commented that the table discussions needed these additional perspectives to better conceptualize the range of literacy skills within the performance levels. In addition, the panelists commented that they would have liked to have seen a broader representation of community types (e.g., rural, suburban, urban) reflected in the table discussions because the needs of adult learners and their environments play a factor in program availability and access to various literacy materials represented in NALS. The committee agreed and solicited participation from members of these professional and geographic areas for the September standard setting.

In an effort to provide results that can be fully understood and replicated, this section provides complete results from the July standard setting reported separately by literacy area.

A complete listing of all judgments made by each panelist who reviewed the prose literacy scale at the July standard-setting session is presented in Tables C-2A, C-2B, and C-2C respectively, for Basic, Intermediate, and Advanced. The information included in the table consists of each participant’s bookmark placement for each round, as well as the corresponding scale score.³ The table number and response probability (rp) level used by each panelist are provided, as well as an indication of whether a

given literacy scale was reviewed by the panelist first (i.e., Occasion 1) or second (i.e., Occasion 2).

Figure C-1 illustrates the bookmark placement results on the scale score metric by round and table. The top three graphs present the results for Occasion 1 (Tables 1, 4, and 7), and the bottom three graphs show the results for Occasion 2 (Tables 2, 5, and 8). The lines are differentiated by performance level to indicate panelists’ cut score recommendations: the upward-facing triangles (Δ) indicate the cut score each panelist recommended for the basic literacy performance standard, the asterisks (*) represent the intermediate literacy performance standard, and the downward-facing triangles (∇) indicate the advanced literacy performance standard. The median Round 3 placement for the table for each cut score is indicated by a standalone symbol (Δ, *, or ∇) on the right-hand side of each graph. The numbers below each graph represent the scale scores corresponding to the median basic, intermediate, and advanced literacy values for the given table.

The graphs in Figure C-1 reflect panelist behavior similar to other, published, bookmark standard-setting sessions (Lewis et al., 1998). That is, as the rounds progress, the variability in bookmark placements tends to decrease, resulting in a relative convergence of bookmark location by the end of the third round. As Figure C-1 illustrates, however, convergence did not always happen, given that bookmark placement reflects individual decisions and biases.

Panelists at Tables 1 and 2 used an 80 percent response probability level (rp80); Tables 4 and 5 were assigned an rp level of 67 percent (rp67); and Tables 7 and 8 were instructed to use a 50 percent response probability level (rp50). Across Tables 1, 4, and 7, there was generally more agreement among panelists in the basic and intermediate cut scores at the conclusion of the Round 3, but the final placements of the advanced cut score varied considerably. A somewhat different pattern is seen across Tables 2, 5, and 8. Panelists at Tables 5 and 8 appeared to reach consensus regarding the cut scores for the basic performance level, Table 2 participants achieved consensus on the cut scores for the intermediate level; and Table 5 achieved consensus on the cut score for the advanced level.

Round 3 data from the two occasions were combined and descriptive statistics calculated. This information is reported by rp level for the prose literacy scale in Table C-3. Across performance levels and rp levels, the standard errors were lowest with the 67 percent response probability level.

Panelists at six of the nine tables reviewed NALS items from the document literacy scale. A complete listing of all judgments made by each pan-

elist who reviewed the document literacy scale at the July standard-setting session is presented in Tables C-4A, C-4B, and C-4C.

Figure C-2 shows the bookmark placement results on the scale score metric for each of the three Occasion 1 (top three graphs) and Occasion 2 (bottom three graphs) cut scores by round and table. Panelists at Tables 3 and 1 used rp80, panelists at Tables 6 and 4 used rp67, and panelists at Tables 9 and 7 used rp50. Final bookmark placements for Table 9 are taken from Round 2, due to a data processing in the Round 3 results for that table.

As with prose literacy, the variability of bookmark placements decreased as the rounds progressed. At all of the tables, there appeared to be more agreement with regard to the cut scores for the basic and intermediate performance levels than for the advanced level. Although some convergence in the advanced cut scores was observed as the rounds progressed, the Round 3 bookmark placements are quite disparate.

Summary statistics for the Occasion 1 and Occasion 2 combined data are presented in Table C-5. Unlike the data for prose literacy, the standard error of the mean for document literacy across rp levels and performance levels was lowest for rp50 and highest for rp80.

Panelists at six of the nine tables reviewed NALS items from the quantitative literacy scale. A complete listing of all judgments made by each panelist who reviewed the quantitative literacy scale at the July standard-setting session is presented in Tables C-6A, C-6B, and C-6C. The Occasion 1 (top three graphs) and Occasion 2 (bottom three graphs) bookmark locations and corresponding scale scores reported by each panelist by round and rp level are given in Figure C-3. Panelists at Table 2 and 3 used rp80, panelists at Table 5 and 6 used rp67, and panelists at Tables 8 and 9 used rp50.

Overall, panelists tended to approach consensus on the cut scores for the basic and intermediate performance levels, although this was not true for Tables 3 or 5. Considerable disparity was evident in the cut scores for the advanced level, and this variability was maintained across all three rounds.

Summary statistics on the combined Occasion 1 and Occasion 2 data are given in Table C-7. The standard error was highest in the basic and advanced performance levels for rp67 and in the intermediate performance level for rp80.

The purpose of using the different response probability instructions was to evaluate the extent to which the different response probability criteria influenced panelists’ judgments about bookmark placements. It would be expected that panelists using the lower probability criteria would place their bookmarks later in the ordered item booklets, and, as the probability criteria increase, the bookmarks would be placed earlier in the booklet.

Bookmark placements are converted to scaled scores in two steps. First the item response theory (IRT) model (here, the two-parameter logistic model, or 2PL) is used to calculate the theta value at which an individual would be expected to answer the item correctly at the specified probability level (see equation 3-1 in the technical note to Chapter 3). Then the theta value is transformed to a scale score value using a linear transformation equation.

Typically, the IRT model equation estimates the value of theta associated with a 50 percent probability of correctly answering an item. As described in the technical note to Chapter 3, the equation can be solved for different probabilities of a correct response. Thus, when the response probability value is 67, the theta estimate is the value at which one would have 67 percent chance of answering the item correctly. Likewise, when the response probability is 80, the theta estimates the value at which one would have an 80 percent chance of answering the item correctly. For a given item, the theta values will increase as the response probability moves from 50 to 67 to 80; the scaled scores will similarly increase.

If panelists apply the different response probabilities correctly, they should shift their bookmark placements in such a way that they compensate exactly for the differences in the way the bookmark placements are translated into thetas and to cut scores. That is, ideally, panelists should compensate for the different response criteria by placing their bookmarks earlier or later in the ordered item booklet. If they are compensating exactly for the different instructions, the theta (and scale score) associated with the bookmark placement should be identical under the three different response probability instructions, even though the bookmark locations would differ. Given these expectations for panelists’ implementation of the response probability criteria, we further examined both the bookmark placements and the resulting scaled cut scores.

In the body of the report, we presented the median results for the Round 3 judgments, as it is these judgments that are typically used in determining the final cut scores. Here we examine the Round 1 judgments, as these would be expected to be more independent than those made after group discussions.

In addition, we look at the results separately by occasion. That is, as

shown in the design for the standard setting (see Table C-1), the order in which the literacy areas were assigned to tables of panelists was alternated so that each literacy area was worked on during Occasion 1 by one table and Occasion 2 by another. The panelists worked with the different areas on different days of the standard setting, with time for interaction with other panelists during the evening. We decided that there might be differences in the way the panelists interpreted and implemented the rp instructions on the first occasion, before there was time for conversation with others (despite the instructions that they should not compare their instructions with each other). We therefore examined results for the first occasion and for the two occasions combined.

To examine the extent to which panelists adjusted their judgments based on the different response probability instructions, we first examined the bookmark placements. Tables C-8, C-9, and C-10 present the Round 1 median bookmark placements for the different rp values, separated by Occasion 1 and for the two occasions combined. The median bookmark placements for intermediate and advanced on Table C-8 (prose) demonstrate the expected pattern; that is, the median bookmark placements increased as the rp criteria decreased.

Regression analyses were run to evaluate whether the response probability criteria had a statistically significant effect on bookmark placement. To increase statistical power for detecting differences, the analyses were conducted by combining all of the judgments into a single data set, which resulted in a total of 252 judgments. Because panelists each made multiple judgments, robust standard errors were calculated with clusters at the panelist level for evaluating statistical significance. A series of dummy codes were created to represent each combination of literacy area and performance level. The rp values were maintained in their original numeric form (50, 67, and 80).

This regression resulted in an R² of .91 (p < .001) and a negative coefficient (−.07) for the rp variable, which approached statistical significance (p = .075). This result suggests a tendency for a negative relationship between rp criteria and bookmark placement. That is, as rp criteria increased, bookmark placement tended to decrease (i.e., bookmarks were placed earlier in the ordered item book). On average, over the different literacy areas and performance levels, a coefficient of −.07 for the rp variable means that panelists using the rp80 instructions placed their bookmarks roughly two items earlier than did the panelists using the rp50 instructions. This is the general pattern that one would expect if panelists were implementing the rp instructions as intended, although the next

section shows that the size of the difference is smaller than the ideal adjustment.

Follow-up analyses were run to examine the effect of the rp criteria separately for each combination of literacy area and performance level, which resulted in nine individual regressions (3 literacy areas × 3 performance levels). For these analyses, dummy codes were created to represent the rp50 and rp80 conditions. The coefficients associated with the dummy codes provide an indication of the extent to which the panelists adjusted their judgments according to the response probability instructions. If panelists were appropriately adjusting their judgments, the coefficient associated with rp50 should be positive (bookmark placed later in the ordered item booklet than when rp67 instructions were used), and the coefficient associated with rp80 should be negative (bookmark placed earlier in the ordered item booklet than when rp67 instructions were used).

Tables C-11, C-12, and C-13 present the results for Occasion 1 judgments (first column) and for Occasion 1 and 2 judgments (second column), respectively, for prose, document, and quantitative literacy. For Occasion 1, seven of the nine rp50 coefficients are positive, and five of the nine coefficients for rp80 are negative, although very few of the coefficients are statistically significant, even at the significance level of p <.10. Similar results are evident for Occasion 1 and 2 combined: seven of the nine rp50 coefficients are positive, and four of the nine rp80 coefficients are negative. Overall, these results show a statistically weak trend in the direction of the correct adjustment to the different rp conditions.⁴

Regressions were run in a similar fashion when the dependent variable was the scaled cut score. The resulting coefficient for the rp criteria was 1.33, which was statistically significant (p < .001). The value of this coefficient suggests a positive relationship between the rp criteria and scaled cut scores; that is, as rp value increases, so do the cut scores. If it were the case that the panelists were insensitive to the rp instructions—making the same bookmark placements on average in all three rp conditions—a positive relationship between the rp condition and the scaled cut scores would result

simply from the effect of the rp condition on the equations used to transform the bookmark placements into the corresponding scale scores. The preceding section shows that the panelists were not insensitive to the rp conditions, however, making adjustments that tended in the correct direction with borderline statistical significance. Given the strong relationship between the rp condition and the scaled cut scores, however, it is clear that the size of the adjustment made by the panelists falls short of the ideal.

As before, a series of follow-up regressions were run, one for each combination of literacy area and performance levels. Dummy codes were again created to represent rp50 and rp80 conditions. If panelists were appropriately adjusting their judgments to compensate for the different response probability instructions, the scale score associated with the bookmark placements should, ideally, be identical under the three conditions. For these analyses, the focus is on the statistical significance of the coefficients; that is, ideally, the coefficients associated with the two rp conditions should not be statistically significant.

Tables C-14, C-15, and C-16 present the results for Occasion 1 judgments (first column) and for Occasion 1 and 2 judgments (second column), respectively, for prose, document, and quantitative literacy. For Occasion 1, four of the nine rp50 coefficients are statistically significant, and five of the nine rp80 coefficients are statistically significant. For Occasions 1 and 2 combined, four of the nine rp50 coefficients are statistically significant, and six of the nine rp80 coefficients are statistically significant. These results suggest a strong relationship between the rp condition and the scaled cut scores.

The committee wishes to acknowledge the assistance and contributions of individuals who served as panelists for the bookmark standard setting and provided valuable input on the performance-level descriptions. The complete list of participants appears at the end of Appendix D.

July Standard-Setting Session

Item

Page

Bookmark Standard-Setting Session for the National Adult Literacy Survey (NALS)

National Research Council, Washington, DC

July 16-19, 2004

Friday, July 16, 2004—The Day Before the Standard-Setting

Saturday, July 17, 2004—Day 1 of Standard-Setting

Sunday, July 18, 2004—Day 2 of Standard-Setting

Monday, July 19, 2004—Day 3 of Standard-Setting

Bookmark Standard-Setting Session for NALS

July 17-19, 2004

National Research Council, Washington, DC

Please answer the following questions in order for us to better understand the characteristics of our group of standard-setting participants.

1. Do your professional responsibilities include direct or managerial responsibilities for the education of adults?

_____ No. Please characterize your professional responsibilities:

_____ Yes. For how many years have you had such responsibilities?

If you answered ‘yes’ to question 1, please answer the following questions:

2. I am involved in adult education in the following roles (please check all that apply):

_____ I am directly involved as an instructor

_____ I am involved in a managerial capacity

3. How would you characterize the educational setting for these adults (check all that apply):

4. How familiar were you with the National Adult Literacy Survey (a.k.a. NALS) before your participation in the standard-setting activities?

_____ Unfamiliar _____ Somewhat familiar _____ Very familiar

Please tell us about yourself (optional):

Items in your booklet are ordered from easiest to most difficult. The easiest items can be answered correctly with a probability of .50 (i.e., 50 percent of the time) by the most people. The most difficult items can be answered correctly with a probability of .50 by the least number of people. In careful consideration of each performance-level description and each item’s difficulty, your task is to identify those skills (represented by items) that you expect persons in each literacy performance level to answer correctly with a probability of at least .50.

First, to establish your Basic Literacy performance level:

Next, to establish your Intermediate Literacy performance level:

Finally, to establish your Advanced Literacy performance level:

Items in your booklet are ordered from easiest to most difficult. The easiest items can be answered correctly with a probability of .67 (i.e., 67 percent of the time) by the most people. The most difficult items can be answered correctly with a probability of .67 by the least number of people. In careful consideration of each performance-level description and each item’s difficulty, your task is to identify those skills (represented by items) that you expect persons in each literacy performance level to answer correctly with probability of at least .67.

First, to establish your Basic Literacy performance level:

Next, to establish your Intermediate Literacy performance level:

Finally, to establish your Advanced Literacy performance level:

Items in your booklet are ordered from easiest to most difficult. The easiest items can be answered correctly with a probability of .80 (i.e., 80 percent of the time) by the most people. The most difficult items can be answered correctly with a probability of .80 by the least number of people. In careful consideration of each performance-level description and each item’s difficulty, your task is to identify those skills (represented by items) that you expect persons in each literacy performance level to answer correctly with probability at least .80.

First, to establish your Basic Literacy performance level:

Next, to establish your Intermediate Literacy performance level:

Finally, to establish your Advanced Literacy performance level:

Bookmark Standard-Setting Session for the National Adult Literacy Survey (NALS)

National Research Council, Washington, DC

July 17-19, 2004

Thank you for participating in the standard-setting meeting for the National Adult Literacy Survey (NALS). In order to help improve future standard-setting meetings, please complete the following questionnaire about your experiences this weekend.

1. How satisfied were you with the advance information given to you about the standard-setting meeting (e.g., memos with information about the hotel, location of the meeting)?

2. How satisfied were you with the food provided during the meeting?

3. How satisfied were you with your hotel accommodations?

4. How satisfied were you with the training you received on Saturday morning?

5. How satisfied were you with the room assignments and table discussions?

6. How satisfied were you with the contributions and guidance of the table leaders?

7. How satisfied were you with the organization of the standard-setting meeting?

8. How satisfied were you with the cut scores decided by your table?

9. How valuable do you feel your contribution was to the outcomes of the standard-setting meeting?

10. How valuable was your participation in the standard-setting meeting to you?

Please feel free to add additional suggestions or comments about the standard-setting meeting.

Thank you!

TABLE C-4 FOLLOWS

TABLE C-6 FOLLOWS