Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors (2021)
Chapter: Appendix A: Search Strategy and Data Coding
Visit
NAP.edu/10766
to get more information about this book, to buy it in print, or to
download it as a free PDF.
Previous Chapter:
References
Appendix A
Next Chapter: Appendix B: Committee and Staff Biographies
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Appendix A
Search Strategy and Data Coding
The committee sought to assemble a set of studies that represented the extent of available evidence. The review focused on literature and programs centered around three categories of outcomes: (1) affective (such as interest, identity/belonging, motivation, and self-efficacy); (2) cognitive (such as knowledge and skills); and (3) behavioral (such as engagement, persistence, and retention). A search was conducted through Scopus requesting studies from the past two decades (2000–2020) and limited to English. Table A-1 shows the syntax used to identify studies. The search parameters were set to be broad and inclusive, which yielded a large number of studies (n = 2,974).
The list was culled down by removing studies that did not meet committee-determined standards for quality and relevance. Specifically, studies were classified based on the following attributes: goal, intervention type, study type, findings reported, outcome category, STEM field, methods, setting, authentic, and the age/grade range. To determine the type of study, the committee used the common guidelines for education research and development (Earle et al., 2013). Box A-1 briefly documents the guide and summarizes the different study types. From this analysis, 309 studies were identified as potentially relevant and requiring additional review.
The committee then reviewed the abstracts for each of the 309 studies. Table A-2 illustrates the systematic culling down of studies that were reviewed at each stage, whereas Table A-3 highlights the ones described in the chapter on outcomes. In Table A-2, studies with measurable impact estimates are those studies that can be characterized as Impact and Effectiveness Research based on Box A-1.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
TABLE A-1 Search Syntax for Locating Studies
Criteria and Syntax |
|---|
| Computing focus |
| (STEM |
| STEM OR science OR technology OR engineering OR mathematic* OR computing OR “computer science” OR “computer program*”) |
| AND Personalized learning focus |
| (“hands-on” OR “design based” OR “maker ed” OR “maker space” OR “place based” OR authentic OR “technology education” OR “project based” OR “problem based” OR “activity design” OR “maker movement*” OR “problem solving” OR “game design”) |
| AND Setting |
| (“in school” OR “out of school” OR formal OR informal OR librar* OR museum* OR “community setting*” OR YMCA OR “boy scout*” OR “girl scout*” OR club* OR “summer camp”) |
| AND Target youth |
| (“K–12” OR “elementary school*” OR “middle school*” OR “high school*” OR child* OR youth* OR adolescent* OR teenager* OR “young student*”) |
| (student* AND NOT {college* OR undergraduate* OR graduate* OR university}) |
| AND Focal outcomes |
| (engagement OR {time on task} OR “future course*” OR “future class*” OR “future club*” OR persistence OR “choice major” OR “curriculum choice*” OR “class choice” OR “career choice”) |
| (interest OR awareness OR attitude*) |
| (knowledge OR skill* OR disposition* OR network*) |
| AND Recency |
| PUBYEAR AFT 2000 |
| Citations returned = 2,974 |
*Focal outcomes were aligned with commonly stated program goals.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
| Search Syntax Strategies and Yield | ||||||||
|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| X | X | X | ||||||
| X | X | X | X | X | X | |||
| X | X | X | X | X | X | X | X | X |
| X | X | X | X | X | X | X | X | X |
| X | X | X | ||||||
| X | X | X | ||||||
| X | X | X | ||||||
| X | X | X | ||||||
| X | X | X | ||||||
| X | X | X | X | X | X | X | X | X |
| 172 | 74 | 154 | 382 | 179 | 331 | 655 | 225 | 802 |
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
TABLE A-2 Search Summary
| Outcome Domain | Abstracts Retrieved & Reviewed | Studies Retrieved & Reviewed | Studies Referenced in the Text | Studies with Credible Impact Estimate |
|---|---|---|---|---|
| Affective | 107 | 76 | 24 | 3 |
| Cognitive | 149 | 102 | 37 | 1 |
| Behavioral | 53 | 40 | 16 | 3 |
| Total | 309 | 218 | 77 | 7 |
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
TABLE A-3 Evaluation Studies Identified Through the Literature Search
| Author (Date) | Study Type | Intervention/Practice |
|---|---|---|
| Affective Outcomes | ||
| Barker et al. (2018) | Exploratory | Wearable technologies |
| Bugallo and Kelly (2014) | Exploratory | Participatory STEM activities |
| Bugallo et al. (2015) | Exploratory | Participatory STEM activities |
| Choudhury et al. (2010) | Exploratory | Participatory STEM activities |
| Denault et al. (2008) | Exploratory | Participatory CS activities |
| Doerschuk et al. (2013) | Exploratory | Participatory CS activities |
| Evans and Schares (2017) | Exploratory | Makerspaces/activities |
| Gardner-McCune et al. (2013) | Exploratory | Participatory CS activities |
| Harriger et al. (2012) | Exploratory | Participatory CS activities |
| Jagiela et al. (2018) | Exploratory | Participatory STEAM activities |
| Krayem et al. (2019) | Exploratory | Participatory STEM activities |
| Ladeji-Osias et al. (2018) | Exploratory | Participatory STEM activities |
| Monterastelli et al. (2008) | Exploratory | Participatory STEM activities |
| Nugent et al. (2016) | Exploratory | Participatory STEM activities |
| Nugent et al. (2019) | Exploratory | Wearable technologies |
| Stapleton et al. (2019) | Exploratory | Participatory STEM activities |
| Ahn et al. (2014) | Design & Development | Participatory STEM activities |
| Erete et al. (2016) | Design & Development | Participatory STEM activities |
| Jin et al. (2018) | Design & Development | Participatory CS activities |
| Lau et al. (2009) | Design & Development | Wearable technologies |
| Pinkard et al. (2017) | Design & Development | Participatory STEM activities |
| Scott and White (2013) | Design & Development | Participatory CS activities |
| Klopfer et al. (2004) | Impact & Effectiveness | Participatory STEM activities |
| Loksa et al. (2016) | Impact & Effectiveness | Participatory CS activities |
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
| Setting | Grade/Age Group | Outcomes for Impact Estimates |
|---|---|---|
| After-school, on site | Elementary; Middle school | Self-efficacy |
| Summer program/camp | High school | Identity; Self-efficacy |
| Summer program/camp | High school | Interest; Motivation; Self-efficacy |
| Summer program/camp | High school | Identity |
| Summer program/camp | High school | Interest |
| Summer program/camp | High school | Interest |
| University | K–12 | Interest |
| University | High school | Interest |
| Summer program/camp | High school | Interest |
| University | High school; Middle school | Interest; Self-efficacy |
| University | High school | Interest |
| Summer & Saturday Program | Middle school | Engagement; Interest |
| University | High school | Interest |
| Camps, Clubs & Competitions | Middle school | Self-efficacy |
| Classroom—traditional & after-school | Elementary school (K–5) | Self-efficacy; Programming design & knowledge |
| University; Summer camp | Middle school | Self-efficacy |
| After-school, on site | Middle school | Identity |
| Community—other | Middle school | Identity |
| Summer program/camp | High school | Knowledge & Skills; Motivation |
| Summer program/camp | Middle school | Knowledge & Skills; Motivation; Interest |
| Multiple OST | Elementary school (K–5) | Identity |
| Multiple OST | High school | Motivation |
| Classroom—traditional | Middle school; High school | Motivation |
| Summer camp | High school | Self-efficacy |
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
| Author (Date) | Study Type | Intervention/Practice |
|---|---|---|
| Cognitive Outcomes | ||
| Bicer et al. (2017) | Exploratory | Makerspaces & activities |
| Brooks and Sjöberg (2019) | Exploratory | Games |
| Bugallo et al. (2015) | Exploratory | Participatory STEM activities |
| Dasgupta et al. (2019) | Exploratory | Makerspaces & activities |
| de Paula et al. (2018) | Exploratory | Participatory STEM activities |
| Denault et al. (2008) | Exploratory | Participatory STEM activities |
| Folk et al. (2015) | Exploratory | Unplugged STEM activities |
| Freina et al. (2019) | Exploratory | Games |
| Gardeli and Vosinakis (2019) | Exploratory | Mobile apps |
| Nugent et al. (2013) | Exploratory | Participatory STEM activities |
| Nugent et al. (2016) | Exploratory | Participatory STEM activities |
| Nugent et al. (2019) | Exploratory | Wearable technologies |
| Ouyang et al. (2018) | Exploratory | Participatory CS activities |
| Sharma and Ali (2018) | Exploratory | Participatory STEM activities |
| Weitze (2017) | Exploratory | Making games/sims |
| Jenson et al. (2018) | Design & Development | Course |
| Yin et al. (2019) | Design & Development | Makerspaces/activities |
| Garneli et al. (2015) | Impact—Efficacy | Games |
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
| Setting | Grade/Age Group | Outcomes for Impact Estimates |
|---|---|---|
| Summer program | High school | Perception of skills & creativity |
| Research Lab setting | Elementary school (K–5) | Interactions with technology, design, decision making |
| Summer program | High school | Engineering knowledge |
| Classroom—traditional | Middle school | Technology & science knowledge; Experimental strategies & graph interpretation |
| After School Program—at a school | Middle school | Computational & game development skills |
| Summer program/camp | High school | Computer programming |
| Classroom—traditional | K–12 (across grade bands) | CT concepts & terms; Recognition of CT |
| Classroom—traditional | Elementary school (K–5) | Programming & documentation skills |
| Classroom—traditional | Elementary school (K–5) | Problem-solving, algorithmic concepts & game design |
| After school, on-site | Middle school | Computer programming, engineering & design, GPS & science, math, robotics & workplace skills |
| Camps, clubs & competitions | Middle school | Computer programming, engineering & design, GPS & science, math, robotics & workplace skills |
| Classroom—traditional | Elementary school (K–5) | Knowledge of circuity & programming; engineering design |
| Classroom—traditional | Elementary school (K–5) | CT concepts |
| Classroom—traditional | High school | Cartesian coordinate system; Applied geometry |
| Classroom—traditional | High school | Disciplinary content; Game design & making |
| Classroom—traditional | Middle school | Computer programming; Academic achievement |
| Summer academy | Middle & High school | Physics & engineering, CT definition & CT skills |
| ns | Middle school | Problem-solving, programming & use of CS constructs |
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
| Author (Date) | Study Type | Intervention/Practice |
|---|---|---|
| Behavioral Outcomes | ||
| Amo et al. (2019)—study 2 | Exploratory | Hands-on cyber security workshops |
| Freudenthal et al. (2011) | Exploratory | Media-propelled CT |
| Gardeli and Vosinakis (2019) | Exploratory | Game: Mobile augmented reality |
| Klein (2013) | Exploratory | Casual mobile game |
| Simpson et al. (2017) | Exploratory | Computer-supported collaborative learning |
| Vickery (2014) | Exploratory | Web design workshop |
| Folk et al. (2015) | Exploratory | Deliberate instruction in CT & algorithm design (unplugged) |
| Weston et al. (2019) | Exploratory | AP and computing courses |
| Charlton and Poslad (2016) | Exploratory (Descriptive) | Maker events: Shareable wearables |
| Mystakidis et al. (2014) | Development | Immersive multi-media learning experience |
| Amo et al. (2019)—study 1 | Impact & Effectiveness (RCT) | Cyber-networking workshop |
| Klopfer et al. (2004) | Impact & Effectiveness (RCT) | Palm versus wearable supported participatory simulations |
| Wanzer et al. (2020) | Impact & Effectiveness | Web-based introductory computing experience |
NOTE: ns means not specified. For each outcome category, the articles are organized by study type and then alphabetized.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
| Setting | Grade/Age Group | Outcomes for Impact Estimates |
|---|---|---|
| Summer camp | 13–17-year-olds | Cyber security engagement & self-efficacy |
| High school math & college calculus class | High school students First year college students | Enrollment in CT courses, engagement in problem-solving, math course enrollment& college grades |
| Primary school classrooms | Age 9–10 | Enjoyment & collaboration |
| Play Club | 10–15-year-olds | Level of emersion & reflection |
| Summer camp | Rising high school freshmen | Positions of power |
| Public library | 8–16-year-olds | Sustained enjoyment |
| K–12 STEM classes | Middle & High school instructors & students. | Recognition & definition of algorithms; engagement in CT |
| High school | High school students | Persistence in computing post high school & technology majors |
| Community setting | Age 14–15 | Engagement in design and development & product application |
| Library | Grades 2–6 | Interest in books & positive mentality toward reading |
| Museum | Middle school (low-income) | General problem solving, science engagement, cyber & digital awareness |
| Science classes | Study 1: 14–16-year-olds Study 2: 11–13-year-olds | Engagement in collaborative problem-solving, motivation to participate in problem-solving & self-reported learning |
| Workshops; summer camps, traditional classes | Middle school; High school | Intent to persist in computing |
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
This page intentionally left blank.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Suggested Citation:
"Appendix A: Search Strategy and Data Coding." National Academies of Sciences, Engineering, and Medicine. 2021. Cultivating Interest and Competencies in Computing: Authentic Experiences and Design Factors. Washington, DC: The National Academies Press.
doi: 10.17226/25912.
Subscribe to Emails from the National Academies
Stay up to date on activities, publications, and events by subscribing to email updates.