Appendix B

Development of the Compendium

The Education Development Center (EDC) was tasked with identifying and reviewing promising, evidence-based, Pre-K–12 science, technology, engineering, and mathematics (STEM) education programs¹ that have scaled and demonstrated evidence of impact, and then listing these programs in a compendium that would inform the Pre-K–12 STEM Education Innovations consensus report. Programs were identified through a self-nomination process and then reviewed and coded by the EDC team.

Since scale and impact can mean different things in different contexts, EDC intentionally kept broad definitions for these two terms. Although the committee provides a more nuanced definition of scale in the report, for the purpose of this compendium, EDC defined scale as reaching at least five to ten times the program’s original population. Impact was defined by EDC as evidence of success from an independent party such as having undergone an evaluation by an external evaluator or a peer review process.

To carry out this compilation work, EDC not only identified innovations by canvasing existing syntheses but also secured nominations from relevant stakeholder groups and interviews with researchers and practitioners. This multipronged approach enabled EDC to distill themes and promising practices as well as identify gaps, challenges, and contexts, which allowed for a focus on how to adapt innovations to meet the needs of different contexts.

___________________

¹For the purpose of this document, EDC uses the term “program” to refer to anything intended to support learning in Pre-K–12 settings (e.g., activities, curricula, tools, professional development).

METHODOLOGY

EDC conducted a seven-step process with the goal of identifying promising Pre-K–12 STEM education programs that have gone through a scaling-up process and demonstrated evidence of impact. In addition to highlighting promising programs, EDC also aimed to identify factors that appear to support programs in their scaling efforts and those that may hinder them, as well as to identify gaps in our understanding of how programs scale.

Review the Literature

First, the EDC team conducted a review of the literature to guide their initial understanding of the elements that may support or negatively impact scale-up of Pre-K–12 STEM educational programs and identify gaps in their understanding of scaling. EDC systematically searched Google Scholar and EBSCO using various combinations of the following keywords: STEM, science, mathematics, engineering, education, innovation, scaling, and scale-up. Results were filtered to identify literature published in the last ten years.

Set Parameters

Based on the review, EDC began the search for relevant Pre-K–12 STEM programs by setting five general parameters to determine inclusion in the compendium. These parameters stipulated that, in order to be included in the compendium programs must

• Occur within the Pre-K–12 grade band;
• Focus on the traditional STEM disciplines (i.e., science, technology, engineering, and mathematics), which could include computer science and data science;
• Be connected to formal classroom teaching and learning (e.g., curricula, professional development);
• Have evidence of scaling (defined as approximately five to ten times the original population or more) though this was not a strict requirement; and
• Have completed a formal external evaluation process to measure their impact and outcomes (i.e., external evaluation or other peer review process).

Develop Program Nomination Form

A Google form was created for potential programs to self-nominate. The form requested general information (e.g., program name and point of contact);

program information (e.g., Pre-K–12 grade band, subject domain[s], intended audience[s]); evidence of scale; and evidence of impact. Evidence of scale and evidence of impact were collected mostly as open-ended responses to allow programs maximum flexibility to explain how they view these parameters and apply them in their individual contexts. A link to this form was included on EDC’s project website and in outreach emails.

Identify Programs

EDC developed a working list of sources and field experts to assist in identifying programs to directly contact regarding the self-nomination process. Based on this, EDC identified programs through the following:

• Professional connections internal and external to EDC;
• Recommendations from the Board on Science Education and the committee;
• Recommendations from relevant organizations (e.g., National Science Foundation [NSF], Education Innovation and Research);
• Federal program databases for programs meeting the criteria that received an award in the past seven years (e.g., NSF and Department of Education [ED] award search databases, the Institute of Education Sciences’ What Works Clearinghouse database); and
• Additional projects discovered throughout the process (e.g., through literature review).

In order to identify specific programs to reach out to, EDC first solicited suggestions from their personal contacts as well as through conversations with program officers from the ED, NSF, National Aeronautics and Space Administration, and National Oceanic and Atmospheric Administration. This resulted in 41 programs identified.

EDC then utilized publicly available databases to review federally funded projects from the past 15 years. This included programs funded by Education Innovation and Research scale-up grants from ED, as well as Innovative Technology Experiences for Students (ITEST) and Teachers and Discovery Research Pre-K–12 (DRK–12) grants from the NSF. EDC included a search for the terms “scale” or “scale-up” in order to limit the results to those referencing scale. The list of programs was then reviewed to include only those programs that had scaled, were focused on a field of STEM, and were connected to formal classrooms. EDC also reviewed the What Works Clearinghouse to identify programs that met criteria. This resulted in the identification of an additional 81 programs.

Finally, EDC’s review of the literature identified an additional 14 programs that had not already been identified through other means.

TABLE B-1 Individual Programs Invited to Self-Nominate

In total, EDC sent invitations to 136 programs inviting them to self-nominate. These programs are shown in Table B-1 (programs as listed in tabular form to save space).

In addition to reaching out to specific programs, EDC also sent information about this opportunity to organizations and asked them to share it with

TABLE B-2 Organizations Contacted

programs in their networks that may be eligible for inclusion in the compendium. In total, EDC contacted individuals representing 64 organizations. These organizations are listed in Table B-2.

Inviting Programs to Self-Nominate

Program contacts received an outreach email encouraging them to nominate their programs and share the opportunity with their networks. EDC also utilized social media for outreach with posts shared by EDC, CADRE, and STELAR. Moreover, EDC sought word-of-mouth recommendations from knowledgeable colleagues and others in the field whenever possible. An initial deadline for nominations was set for May 1, 2024, and then extended to May 24.

Review and Analyze Nominations

In total, 65 nominations were received (see Table B-3).

Descriptive characteristics of the nominated programs were analyzed using Excel. A rubric was developed to help organize and guide EDC’s review of nominated programs (see Table B-4).² First, nominations were reviewed to determine to what extent there was evidence of the following factors that may contribute to successful scaling efforts (which was informed by the literature search):

• a core program with room for adaptation to different contexts
• alignment with policies, goals, or standards
• building individual or organizational capacity
• partnerships and networks

Additionally, nominations were reviewed to determine evidence of scale and impact in four areas:

• fidelity during scaling (scale)
• scaling to new audiences or contexts (scale)
• participant outcomes were achieved or skills developed (impact)
• understanding of effectiveness for different contexts/learners (impact)

For each element in the rubric, programs were categorized as providing no evidence, some evidence, or strong evidence.

___________________

²Although EDC staff are involved with some nominated programs, these individuals were not involved in the review of nominated programs.

TABLE B-3 List of Nominations Received

Notes: Programs highlighted in blue were excluded because they do not have connections to the Pre-K–12 setting, programs highlighted in green were excluded because they do not have external evaluations or peer review, programs highlighted in orange were excluded because they do not connect to STEM disciplines.

TABLE B-4 Rubric

Summarize Programs

Summaries for all programs are provided in Appendix C. Once all nominations were assessed using the rubric, programs were reviewed to create summaries, including descriptions of scaling efforts and evidence of success.

LIMITATIONS TO THE APPROACH

The process laid out above provided opportunities for EDC to identify and reach out to a wide range of Pre-K–12 education programs that have gone through a scaling-up process and demonstrated evidence of impact. But it is also important to consider some limitations to this approach; these include:

• The literature review was meant to be informative and does not provide an exhaustive review of the literature.
• Although EDC did specific outreach to programs in addition to broad communication about the opportunity, the review is limited to the nominations received. Therefore, there are likely other promising programs that are not included in the compendium.
• EDC’s assessment of programs was limited to the information provided on the nomination form and what could be reviewed on each program’s website. Therefore, the review of individual program may not fully reflect the reality of certain programs.
• The EDC team designed and used the rubric to organize descriptive summaries of the nominations. It is not meant to be used to officially classify the success of each program or to compare the nominated programs to each other.
• Nominated programs are implemented in complex educational environments and depend on several additional factors such as policies, district leadership, curricular goals, and morale and support in the educational environment. Therefore, programs and findings highlighted should be viewed as only one component in a complex system of factors that influence implementation in various settings.