Evaluation of the Voluntary National Tests: Phase 1 (1999)
Chapter: Appendix G: Observations of Cognitive Labs and Bias Reviews
Appendix G Observations of Cognitive Labs and Bias Reviews
|
Activity |
Observations |
Purpose of Observation |
|
Cognitive Lab Training |
||
|
Palo Alto, CA (May 11–12, 1998) |
Participated in training provided to cognitive lab interviewers. Training consisted of two days of overview, tape observation, role playing, and discussion. |
To acquaint evaluators with the interview methodology used in the cognitive labs. Also, attending both sessions provided evaluators with the ability to observe the evolution of the training materials. |
|
East Lansing, MI (April 20–21, 1998) |
|
|
|
Live Cognitive Lab Interviews |
||
|
AIR, New England (May 26, 1998) |
M2 (2 taped) |
To allow evaluators to observe the cognitive lab process and to determine whether the cognitive labs would provide information that could improve the quality of items. Observing interviews at several locations allowed evaluators to determine interviewer quality and whether the interviews were being conducted consistently across sites. |
|
San Antonio, TX (June 8–9, 1998) |
R9 (1 taped) |
|
|
AIR, Palo Alto (June 10–12, 1998) |
R7 (1 live interview, 1 taped) |
|
|
East Lansing, MI (June 22–23, 1998) |
R16 (2 live interviews) |
|
|
|
R4 (4 taped) |
|
|
|
R9 (4 taped) |
|
|
|
M7 (1 live interview, 4 taped) |
|
|
|
M10 (2 live interviews, 1 taped) |
|
|
|
R20 (1 live, 1 taped) |
|
|
|
M11 (1 live interview) |
|
|
|
M19 (1 live interview) |
|
|
|
M16 ( live interviews) |
|
|
|
R22 (1 taped) |
|
|
|
R18 (1 taped) |
|
|
|
R22 (1 taped) |
|
|
Activity |
Observations |
Purpose of Observation |
|
Taped Cognitive Lab Interviews |
||
|
AIR, Washington, D.C. (July 1–2, 1998) |
M2, M10 (9 tapes [some of which had already been seen] and 1-on-1 forms) R4, R9 (9 tapes [some of which had already been seen] and 1-on-1 forms) |
To allow evaluators to focus on the contribution of the cognitive labs to item quality for a small sample of items and to allow evaluators to review the 1-on-1 forms filled out by the AIR interviewers. |
|
Bias and Sensitivity Review |
||
|
Math—San Antonio, TX (July 6–7, 1998) |
Panel members reviewed 1,400–1,500 items. |
To determine the impact of bias and sensitivity review on item quality. |
|
Reading—Chicago, IL (July 6–7, 1998) |
Panel members reviewed 350–400 passages. |
|
