Evaluation of the Voluntary National Tests: Phase 1 (1999)
Chapter: Appendix F: Revised Item Development and Review Schedule for VNT
Appendix F Revised Item Development and Review Schedule for VNT.
Below is a summary of the key dates in 1998 for delivery of items, NAGB review, and panel meetings:
|
Date |
Activity |
|
8/18–25 |
Achievement level review panels |
|
8/18–22 |
Content expert panels |
|
9/1–4 (tent.) |
Content coverage panels |
|
9/28 |
1st batch of math items to NAGB |
|
10/5 |
1st batch of reading items to NAGB |
|
10/16 |
Reviews back from NAGB |
|
10/30 |
Final batches of items to NAGB |
|
11/13 |
Final reviews back from NAGB |
|
11/19–21 |
NAGB final action on items |
The following chart summarizes the review panels, tentative meeting dates, purpose, and composition:
|
Review Panel |
Tentative Dates |
Purpose |
Composition |
|
Ongoing Reading Expert Panel (ORE) |
8/18-8/20 |
To review items, using information from cognitive labs, with the goal of providing final recommendations to the publishers |
Content experts, including teachers and curriculum specialists with experience in elementary school reading instruction |
|
Ongoing Math Expert Panel (OME) |
8/20-8/22 |
|
Teachers and curriculum specialists with experience in middle school math instruction |
|
Independent Reading Achievement Levels Panel (IRAL) and Independent Math Achievement Levels Panel (IMAL) |
Early September |
To ensure that the NAEP achievement levels are well covered by the VNT items |
Teachers, curriculum experts and teacher educators familiar with NAEP achievement levels and content areas of math and reading |
|
Independent Reading Content Coverage Panel (IRCC) and Independent Math Content Coverage Panel (IMCC) |
First week of September |
To ensure that the NAEP content frameworks are well covered by the VNT items |
Teachers, curriculum experts, and teacher educators familiar with NAEP content frameworks |
The following table summarizes the revised plans for VNT item development:
|
Month |
Date |
Activity |
|
July |
7/27–28 |
• Readability study held at AIR |
|
|
7/27–31 |
• Identify ORE and ON-IE Panel members |
|
|
|
• Plan expanded item review activities, including budget |
|
|
|
• Identify ORE and OME Panel members |
|
|
|
• Arrange for meeting space |
|
|
|
• Determine review sequence for items (e.g., ECRs first in math, intertextual sets first in reading) |
|
|
Late July |
• SAC #1 reviews reading items via teleconference |
|
August |
8/3–7 |
• Receive all reading and math items in their most current state from publishers |
|
|
|
• Contact IRAL, IMAL, IRCC, and IMCC Panel members |
|
Month |
Date |
Activity |
|
|
|
• Prepare items and related cognitive lab data in preparation for OME and ORE Panel meetings |
|
|
|
• Develop OME and ORE Panel meeting agendas |
|
|
|
• Design format/process for providing recommendations to publishers from OME and ORE Panel meetings |
|
|
|
• Begin reviewing cognitive lab tapes for rubric development and to inform content panels |
|
|
8/5–6 |
• SAC # 2 reviews math items in person |
|
|
8/6–8 |
• NAGB meeting |
|
|
8/10–14 |
• Finish preparing meeting materials for OME and ORE Panels |
|
|
|
• Begin preparing materials for IRAL and IMAL Panel meetings |
|
|
|
• Begin preparing student work for rubric refinement |
|
|
8/10 |
• Rubric refinement work begins at Harcourt and Riverside |
|
|
8/18–21 (tentative) |
• IRAL Panel meeting |
|
|
8/22–25 (tentative) |
• IMAL Panel meeting |
|
|
8/18–20 (tentative) |
• OME Panel meeting |
|
|
8/20–22 (tentative) |
• ORE Panel meeting |
|
|
8/24–9/4 |
• Process reviews from OME and ORE Panels |
|
|
|
• Send revision requests to publishers as completed |
|
|
|
• Send final recommendations to Harcourt and Riverside by 9/4 |
|
September |
First week of September |
• IRAL and IMAL Panel meetings |
|
|
Mid-September |
• Additional OME and ORE Panel meetings, as needed |
|
|
|
• Revisions to math and reading items processed at AIR |
|
|
|
• Requests for revision sent to Harcourt and Riverside |
|
|
9/21 |
• First batch of revised items due from Harcourt and Riverside |
|
|
9/21–10/2 |
• Check revised items to prepare for NAGB |
|
|
|
• Prepare items for submission to NAGB |
|
|
9/28 |
• Math items submitted to NAGB |
|
|
September – October |
• Additional item development, if needed, based on IRAL, IMAL, IRCC, and IMCC Panel meetings |
|
October |
10/5 |
• Reading items to NAGB |
|
|
October |
• Continue process of reviewing OME and ORE experts' comments on reading and math items |
|
|
|
• Submit requests for revision to publishers |
|
|
|
• Process items as they are returned from publishers |
|
|
10/16 |
• All reviews to items in batch #1 returned from NAGB. |
|
|
10/30 |
• Final batch of math and reading items to NAGB for review (assuming no new item development) |
|
Month |
Date |
Activity |
|
November |
11/2 |
• AIR completes review of newly developed items, if necessary |
|
|
11/3–10 |
• Publishers revise newly developed items, if necessary |
|
|
11/16 |
• Newly developed items to NAGB for review, if necessary |
|
|
11/13 |
• Final NAGB review of items (excluding any newly developed items) |
|
|
11/19–21 |
• NAGB Board meeting and final action on items |
|
|
|
• Final NAGB review of newly developed items |
|
|
11/30 |
• NAGB Reading and Math Committees final action on newly developed items, if necessary |
|
December |
December |
• Pilot test forms assembled |
|
January |
1/1/1999 |
• Camera-ready copy to printers |
