Appendix C

Summary of Data Content Prioritization Process

To investigate the most-needed data content, the panel interviewed experts and stakeholders on the following topics:

• Level of schooling—early childhood education, K–12, higher education, adult education, career and technical education;
• Actors—schools/facilities and administrators, teachers, students, families/parents;
• Characteristics of actors—educational factors, context and conditions, curriculum, instruction, schools and teacher context, students and home context; and
• Outcomes—education outcomes, workforce, other life outcomes.

In those interviews and panel meetings, the panel probed further about:

• Stakeholders;
• Unit of data collection and measures, including:
  • Important outcomes to measure (e.g., educational, social); and
  • Important educational factors, context, and conditions to measure (e.g., disabilities, family socioeconomic status, curriculum, school discipline policy, availability of wraparound services);
• Relevant levels of education (e.g., early education, K–12, postsecondary, lifelong learning);
• Unit of collection/data acquisition (e.g., students, parents, schools/facilities, teachers); and

• Types of products (e.g., in-depth, medium, fast, visualizations), regularity, and intended audiences.

To determine which topics should be given the highest priority, the panel specified four minimum criteria, requiring that each topic meet at least one of the four. These criteria are:

• What sets the (proximal and distal) context for education?
• What constitutes the developmental and social processes and structures of education?
• What are the social and psychological experiences of students, teachers, and administrators in education?
• What are the (interim, short-, and long-term) outcomes of education?

Based on these criteria, the panel identified 112 topics of interest. These topics were then prioritized in two ways: first, and most critically, by the importance or value of the topic, and second, by level of effort required (i.e., whether NCES could reasonably make progress on the topic).

The importance of each topic was evaluated based on the following dimensions:

1. The presence of a legal mandate or restriction;
2. The impact on students and student outcomes, including:
   1. The topic supports understanding of factors affecting student outcomes; and
   2. Analytics result in actionable interventions affecting student outcomes (i.e., factors, areas, and aspects that influence education and educational outcomes and help stakeholders understand how to enhance outcomes of students, assist people, and move the system forward);
3. Balance across life cycle—covers early childhood education and adult education in addition to K–12 and higher education;
4. Sufficient coverage of:
   1. Workers’, teachers’, and administrators’ information linked to learners; and
   2. Institutional information/context linked to learners (this might include the use of online education);
5. Impact on (national or state) policy: whether data on this topic provide information on how education policy and programs relate to other social policies (e.g., housing, labor force, poverty);
6. National importance: economic viability, social cohesion, basic understanding of the preparation of the future workforce; or emerging, isolated, or dispersed topics with broader national implications

6. (for education, for the economy, etc.) that are useful to the education community;
7. Number of interested parties/data users, both key data users (e.g., the White House) and others; and
8. Whether data products have key uses, such as answering key research questions to fill knowledge gaps (e.g., What data do users need? Can data products help decision making?)

NCES’s ability to make progress on each topic was evaluated based on the following:

9. Would data products on this topic fill a gap in existing data, statistical products, reports, or tools?
10. Measurement/operationalization feasibility:
    1. Is the topic easy or difficult to measure? (e.g., student learning [difficult] vs. employment [easier but still difficult] vs. student attributes such as age and gender [easiest]).
    2. Do users need population data or good data with a low standard error? Deep detail? Individual records vs. aggregated data? Longitudinal or intergenerational data?
    3. How uniform are the data definitions collected across the domain on institutions or on people?
11. Would standards set by NCES add value and be worthwhile?
12. Do administrative data “pre-exist,” or would administrative forms and procedures need modification to collect such data appropriately? If data exist, does NCES have access to these data and can the data easily be added to NCES’s operations, via data linkage or otherwise?
13. If NCES collected/acquired the data, would NCES’s work add value or duplicate? What is the return on investment?
14. Can data be delivered to users on time?
15. Would this topic be substantially advanced by NCES’s involvement, such as providing an accessible technological solution, facilitating data linkage, or otherwise supporting education entities’ access to data infrastructure?

For each of these criteria, the 112 topics were assigned a yes/no determination as to whether the topic met each criterion. Allowing for the division of criteria 2 and 4 into two separate dimensions, and of criterion 10 into three separate dimensions, a topic could meet up to 10 criteria on importance and up to 9 criteria on feasibility.

These criteria and the process used are described here both to document how the panel evaluated potential topics for inclusion in NCES’s

data collections and as a possible model for NCES as the Center pursues its strategic planning. No attempt is made to assert that the 19 criteria are equal in value, and a variety of weights might be attached to each criterion; some criteria might be considered so important that satisfying just that criterion would be sufficient justification for including a topic, while others might have lesser importance and be insufficient alone. We suggest using the following strategies. First, if a topic fails to satisfy any of the criteria, one might reevaluate the importance of that topic. Second, the degree to which a topic satisfies multiple criteria can be interpreted as a rough measure of the broad importance or feasibility of the topic. Third, the criteria might be used as tools for identifying gaps in current or planned data collections (e.g., could a data collection be modified to more thoroughly address NCES’s research priorities?). Clearly, there are reasons to limit any data collection (e.g., cost constraints and concerns about response rates), and we are not suggesting that every data collection be turned into a massive effort. Sometimes, a risk in data-collection development is that everyone has a topic to add. Still, there may be data collections whose utility can be greatly increased with only minor changes. Fourth, to counterbalance the third point, there may be topics so thoroughly covered elsewhere that there is little advantage to adding data items on those topics. Finally, we should emphasize that, in addition to data collection, another way for NCES to provide leadership on these topics is by creating standards and tools that others may use. NCES has done this by creating tools such as the Classification of Instructional Programs and the Department of Education’s School Climate Surveys.