A Vision and Roadmap for Education Statistics (2022)
Chapter: Appendix E: Institute of Education Sciences and NCES Product Review Processes
TABLE E-1 NCES Products: Required Reviews
| Product: | Type of Review Required | |||||||
|---|---|---|---|---|---|---|---|---|
| Level 1 internal review | Level 1a rolling review | Level 2. statistica l review | Level 3. AC/CS/OC | Level 4. AC | Level 5. NCES/RIMG/OMB | Level 7. IES external review | Level 7a. IES internal review | |
| Compendium | X | (X) | X | |||||
| Directory | X | |||||||
| NCES Handbook | X | |||||||
| Updated indicators | X | |||||||
| Web Tables | X | |||||||
| Statistical Analysis Report | X | X | X | |||||
| R&D Report | X | (X) | (X) | |||||
| Technical/Methodological Report | X | (X) | (X) | |||||
| Statistics in Brief | X | X | X | |||||
| First Look/Highlights/Data Point | X | X | ||||||
| Re-packaged Excerpts only | X | |||||||
| Guide (e.g., Programs & Plans) | X | |||||||
| Working Papers | X | |||||||
| Data File (including CD ROM/DAS/WEB) | X | |||||||
| Data File Documentation/User’s manuals (must accompany data file) | X | |||||||
| Video/Data | X | |||||||
| Conference Report | X | |||||||
| Non-data Videotape (e.g., conference, Commissione’rs statements) | X | |||||||
| Brochure/Pamphlet | X | |||||||
| Newsletters | X | |||||||
| Co-op Products (e.g., FORUM, NPEC) | (X) | X | ||||||
| Questionnaires | X | |||||||
| Glossaries | X | |||||||
| (x) Review is at the discretion of the specified level | ||||||||
| Level 1. Internal Review | Requires BC/AC/CS/OC review and approval. | |||||||
| Level 1a.Rolling Review | Requires BC/AC/CS review and approval as parts of the whole are completed. Final product requires full Level 1 review. | |||||||
| Level 2. Statistical Review | Requires BC/AC/CS review and approval. | |||||||
| Level 3. AC/CS/OC | Requires BC/AC/CS/OC review and approval. | |||||||
| Level 4. AC | Requires BC/AC review and approval. No official NCES distribution but made available via web or special request. | |||||||
| Level 5. NCES/RIMG/OMB | Requires BC/AC/OC approval within NCES plus review/approval by RIMG & OMB, and copy to Chief Statistician. | |||||||
| Level 7. IES External Review | Requires External comments and review/approval by IES Action Editor and IES DDS | |||||||
| Level 7a IES Internal Review | Requires review/approval by IES Action Editor and IES DDS | |||||||
SOURCE: Reproduction of Table 6-1-A from NCES’s document, “Review Procedures: 6-1 Review of Reports and Data Products.” Available: https://nces.ed.gov/statprog/2012/pdf/Chapter6.pdf [March 2022].
NOTES: This table shows NCES’s standards for its report-review process. In practice, there is some flexibility as staff use their judgment on a case-by-case basis to determine which levels of review are needed.
TABLE E-2 NCES Web Products: Required Reviews
| Product: | Type of Review Required | ||||
|---|---|---|---|---|---|
| Level 1. internal review | Level 2. statistical review | Level 3. AC/CS/OC | Level 4. AC | Level 6. Author/web publisher | |
| Web Applications: NCES Products: (with #) |
|||||
| pdf file | X | X | |||
| Html | X | X | |||
| ASCII/Excel/data base file* | X | X | |||
| Conference Reports/Co-op Products | X | X | |||
| Tools: | |||||
| Locator | X | X | |||
| Peer Tool: Public Access | X | X | |||
| Data Tool./Table and model servers | X | X | |||
| Questionnaire Tool | X | X | |||
| Glossary Search - based on approved product with NCES #) | X | X | |||
| Table/Figure Search | X | ||||
| WEB sites; pages; information sources: | |||||
| Survey/Program site | X | X | |||
| Web Publications | X | ||||
| Quick Facts | X | ||||
| Video | |||||
| Informational Videos | X | X | |||
| Data Videos | X | X | |||
| PowerPoint Presentations | X | X | |||
| Quick tables/figures | X | X | |||
| Co-op Products | X | X | |||
| Working Papers | X | ||||
| *Excludes pre-release data. X All tools with micro data will be subjected to disclosure review, as well as technical review. A full review is required only for new products. Updates to current products only require review of the update information as appropriate. | |||||
| Level 1. Internal Review | Requires BC/AC/CS/OC review and approval. | ||||
| Level 1a.Rolling Review | Requires BC/AC/CS review and approval as parts of the whole are completed. Final product requires full Level 1 review. | ||||
| Level 2. Statistical Review | Requires BC/AC/CS review and approval. | ||||
| Level 3. AC/CS/OC | Requires BC/AC/CS/OC review and approval. | ||||
| Level 4. AC | Requires BC/AC review and approval. No official NCES distribution, but made available via web or special request. | ||||
| Level 5. NCES/RIMG/OMB | Requires BC/AC approval within NCES plus review/approval by RIMG & OMB, and copy to Chief Statistician. | ||||
| Level 6. Author/Web Publisher | Assumes full review as appropriate for the original NCES numbered product. | ||||
SOURCE: Reproduction of Table 6-1-B from NCES’s document, “Review Procedures: 6-1 Review of Reports and Data Products.” Available: https://nces.ed.gov/statprog/2012/pdf/Chapter6.pdf [March 2022].
TABLE E-3 Review Process Durations for NCES Reports: FYs 2019–2021
| IES Internal Review (n = 95) | |||
|---|---|---|---|
| Median | Minimum | Maximum | |
| Time to Complete Review Between the NCES Chief Statistician and NCES Commissioner‡ | 25 d | N/A | 113 d |
| 5 w | 23 w | ||
| 1 m | 5 m | ||
| Time Between Receipt of Report by Office of Science and First Memo Sent to NCES | 5 d | 0 d | 39 d |
| 1 w | 0 w | 8 w | |
| <1 m | 0 m | 2 m | |
| Time Days Between Receipt of Report by Office of Science and Final Approval | 12 d | 0 d | 99 d |
| 2 w | 0 w | 20 w | |
| <1 m | 0 m | 5 m | |
| Time in Office of Science | 8 d | 0 d | 80 d |
| 2 w | 0 w | 16 w | |
| <1 m | 0 m | 4 m | |
| IES External Review (n = 25) | |||
| Median | Minimum | Maximum | |
| Time to complete review between the NCES chief statistician and NCES commissioner‡ | 69 d | N/A | 214 d |
| 14 w | 43 w | ||
| 3 m | 10 m | ||
| Time between receipt of report by Office of Science and disposition memo sent to NCES | 30 d | 15 d | 83 d |
| 6 w | 3 w | 17 w | |
| 1 m | <1 m | 4 m | |
| Time between receipt of report by Office of Science and final approval | 67 d | 26 d | 234 d |
| 13 w | 5 w | 47 w | |
| 3 m | 1 m | 11 m | |
| Without 2 | |||
| outliers:** | |||
| 126 d | |||
| 25 w | |||
| 6 m | |||
| Time in Office of Science (includes time that reports are with external reviewers) | 43 d | 23 d | 147 d |
| 9 w | 5 w | 29 w | |
| 2 m | 1 m | 7 m | |
| Without 2 | |||
| outliers:** | |||
| 84 d | |||
| 17 w | |||
| 4 m | |||
SOURCE: IES document provided to the panel, “Peer Review of IES Reports”; NCES response to question from the panel, pp. 35–36.
LEGEND: This table summarizes the duration of NCES product review at various stages, differentiated by whether the product must undergo IES internal or external review. For instance, NCES reports requiring an IES internal review took a median of 25 business days to complete the NCES internal review process, and then required a median of 12 additional business days between receipt of the report by the Office of Science and final approval.
NOTES: d means number of business days, w means number of calendar weeks (# days/5), and m means approximate number of calendar months (# days/22).
‡Timing information is only available for the chief statistician- and commissioner-review stage of NCES internal review since the iterative nature of the review process prior to this stage makes it challenging to measure timing. Duration information from NCES’s Review Tracking System for FYs 2019–2021 includes 92 reports (instead of 95) undergoing IES internal review and 19 reports (instead of 25) undergoing IES external review. Thus, the summary of timing for the NCES internal review process is based on a significantly overlapping but not completely identical set of reports. These times are in addition to the times for IES internal or external reviews shown in the rows below.
The time in the Office of Science shows, of the total time between when the Office of Science receives a report and when the report is approved, how much of that time reports are in the hands of the Office of Science. For the balance of the total time, reports are with NCES for revision.
**The maximums shown for IES external reviews include two outliers, for which the maximum total days to approval was 234 and 190 days, and the maximum days in the Office of Science was 147 and 107 days. When excluding those outliers, the maximum total days to approval was 126 days and the maximum days in the Office of Science was 84 days. Both outliers were National Assessment of Educational Progress reports.
This page intentionally left blank.
