EXECUTIVE SUMMARY

The ways in which young people learn about science, technology, engineering, and mathematics (STEM) has fundamentally changed in the past decade. More so than ever, young people now have opportunities to learn STEM in a wide variety of settings, including clubs, summer programs, museums, parks, and online activities. They spend more time in supervised programs outside of school, and they have greater access to on-demand learning resources and opportunities. At the same time, STEM learning outside of school has become a focal piece of the education opportunities provided by many national nonprofit organizations, statewide education networks, federal programs, and corporate and family foundations. And there is growing evidence that opportunities to learn STEM outside of school directly affect what is possible inside classrooms, just as what happens in classrooms affects out-of-school learning.

The Committee on Successful Out-of-School STEM Learning was charged with outlining the criteria that policy makers, program developers, and other stakeholders can use to identify effective out-of-school STEM settings and programs. It was also charged with identifying those criteria for which data are readily available and those for which further work is needed to develop appropriate data sources. To address its charge, the committee organized a National Summit on Successful Out-of-School STEM Learning, reviewed relevant research, and commissioned papers to synthesize existing research.

Research and evaluation related to learning outside of school have been conducted by professionals from many fields, including youth development, cognitive and social development, informal learning, and out-of-school time. Evidence from these fields shows that STEM learning results from the dynamic interactions that occur over time among the diverse settings in which learning occurs (e.g., youth groups, hobby clubs, museums, libraries, schools, home), the community and culture in which they are embedded, and the characteristics of the learner (e.g., interests, dispositions, values). Within this dynamic system, out-of-school programs have been shown to:

• contribute to young people’s interest in and understanding of STEM,
• connect young people to caring adults who serve as role models, and
• reduce the achievement gap between young people from low-income and high-income families.

Research and evaluation findings are not yet robust enough to determine which programs work best for whom and under what circumstances. The limitations of the existing research are due to the many types of out-of-school STEM programs and the difficulties of measuring the outcomes of such programs. The findings are strong enough, however, to identify three criteria of programs that produce positive outcomes for learners: they are engaging, responsive, and make connections.

To better understand how productive out-of-school STEM programs contribute to young people’s interest in and understanding of STEM, evaluations must address individual, program-, and community-level outcomes. Building the capacity to generate evidence at these three levels will lead to a clearer picture of how programs affect outcomes across settings and time. In addition, mapping the STEM learning assets of communities can inform decisions about where further investment is needed and support connections among STEM learning opportunities. Innovative measures can illustrate what programs work for whom and under what circumstances and can make it easier to compare and aggregate program outcomes.

As measurement work moves forward, it is important to avoid two common mistakes. One is to depend entirely on short-term student learning outcomes as indicators of productive programs rather than recognizing more complex and varied outcomes. The second is to measure outcomes in ways that alter the nature of productive programs or ignore the differences in out-of-school programs in order to generate comparative or aggregate data.

The committee identified six actions that policy makers, program developers, and stakeholders should take to develop and support productive programs:

• Understand the local conditions for community programs that support STEM learning: Build a map and bridge the gaps.
• Design programs to achieve access, equity, continuity, and coherence: Connect young people to opportunities to learn.
• Support the use of creative and responsive approaches to evaluate the success of programs at the individual, program, and community levels: Support innovative evaluation approaches.
• Increase the professionalization of out-of-school program leaders and staff: Provide professional development.
• Strengthen the STEM learning infrastructure: Build an infrastructure that will last.
• Invest in research to improve our understanding of STEM learning in out-of-school programs: Explore how STEM learning ecosystems work.

Some of these activities can only be undertaken at the local level; some will require national-level involvement. All of them need to be undertaken with sensitivity to the young people who have historically been underserved by STEM learning programs, including girls, ethnic minorities, and those from economically marginalized communities. Together, the actions above can support productive out-of-school STEM programs.