Exploring the Use and Application of Implementation Science in Health Professions Education: Proceedings of a Workshop (2023)
Chapter: Appendix E: Workshop Materials
E
Workshop Materials
Implementation Research Logic Model (IRLM)
IRLM—Implementation Outcomes Worksheet
Implementation outcomes are “the effects of deliberate and purposive actions to implement new treatments, practices, and services” (Proctor et al., 2011). They serve as (1) indicators of implementation success, (2) proximal indicators of implementation processes, and (3) intermediate outcomes in relation to student and clinical/patient outcomes:
NOTE: *IOM Standards of Care
Unlike clinical/patient outcomes, implementation outcomes are often at the level of the system, setting, or educator and typically not at the level of the patient/client. Some outcomes may be measured by researchers, whereas others may be measured through administrative records.
To identify implementation outcomes for your project, it is helpful to work backward from the most downstream/distal/long-term to more upstream/proximal/short-term outcomes.
- For simulation-based education (SBE), what are the student outcomes you are interested in? These may include skills, knowledge, etc. Add these to your IRLM.
- From the list of implementation outcomes starting on the next page, place a checkmark (√) next to ones that may be relevant to teaching students using SBE. Add these to your IRLM.
| √ | Student Outcome | Definition |
|---|---|---|
| Knowledge | Facts and information related to providing clinical care | |
| Self-Efficacy | A student’s belief in their ability to manage a specific type of care (related to the simulation) | |
| Problem Solving | The process of defining a problem; determining the cause of the problem; identifying, prioritizing, and selecting alternatives for a solution; and implementing a solution | |
| Critical Thinking | The ability to identify and analyze problems as well as seek and evaluate relevant information in order to reach an appropriate conclusion |
| Motivation | Students’ desire to engage in the simulation activity and/or complete similar care in a real-world setting | |
| Decision Making | The process of creating a rationale and arriving at an accurate outcome or plan, with a focus on the minimization of errors |
- From the list of implementation outcomes below, place a checkmark (√) next to ones that may be germane to using SBE with students. For each outcome, operationalize it for SBE and add it to your IRLM.
| √ | Implementation outcome | Definition |
|---|---|---|
| RE-AIM Framework (www.re-aim.org; Glasgow et al., 1999) | ||
| Reach | The absolute number, proportion, and representativeness of individuals who are willing to participate in being a student learning with SBE or an instructor using SBE | |
| Effectiveness | The impact of SBE on important outcomes for students, including potential negative effects, quality of life, and economic outcomes | |
| Adoption | The absolute number, proportion, and representativeness of settings and agents (people who deliver the program) who are willing to initiate teaching with SBE | |
| Implementation | At the setting level, implementation refers to the instructor’s fidelity to the various elements of an SBE protocol, including consistency of delivery as intended and the time and cost of teaching with SBE. At the individual level, implementation refers to students’ use of and engagement with SBE | |
| Maintenance | The extent to which SBE becomes institutionalized or part of the routine organizational practices and policies. Within the RE-AIM framework, maintenance also applies at the individual level. At the individual level, maintenance is defined as the long-term effects of SBE on outcomes six or more months following the most recent SBE encounter | |
| Proctor et al., 2011 | ||
| Acceptability | The perception among implementation stakeholders that SBE is agreeable, palatable, or satisfactory | |
| Adoption | The intention, initial decision, or action to try or employ SBE (an evidence-based practice) | |
| Appropriateness | The perceived fit, relevance, or compatibility of SBE for a given classroom setting, instructor, or student; and/or perceived fit of SBE to address a particular issue or problem | |
| Cost | The cost impact of the SBE implementation effort | |
| Feasibility | The extent to which SBE can be successfully used or carried out within a given classroom/department/institution | |
| Fidelity | The degree to which SBE was implemented as it was prescribed in the original protocol or as it was intended by the developers | |
| Penetration/Uptake | The integration of SBE within an educational setting and its subsystems | |
| Sustainability | The extent to which newly implemented SBE is maintained or institutionalized within a setting’s ongoing, stable operation |
The materials below are adapted from Smith et al. (2020) under a Creative Commons Attribution 4.0 International License.
Implementation Research Logic Model (IRLM)
IRLM—Determinants of Implementation Worksheet
Determinants of implementation are constructs that have been associated with effective implementation. Often, researchers think of determinants as implementation barriers and facilitators, but they can also be mediators, moderators, predictors, and/or outcomes. One of the most comprehensive lists of determinants comes from the Consolidated Framework for Implementation Research (Damschroder and Hagedorn, 2009).
- From the list of CFIR constructs below, place a checkmark (√) next to ones that may be germane to studying SBE in your classroom/department/institution. It is important to capture all factors that may affect the implementation of SBE.
- For each determinant, operationalize it for your context and add it to your IRLM.
| √ | Determinant | Definition |
|---|---|---|
| Intervention Characteristics | ||
| Intervention source | Perception of key stakeholders about whether SBE is externally or internally developed | |
| Evidence strength and quality | Stakeholders’ perceptions of the quality and validity of evidence supporting the belief that SBE will have desired outcomes | |
| Relative advantage | Stakeholders’ perception of the advantage of implementing SBE versus an alternative solution | |
| Adaptability | The degree to which SBE can be adapted, tailored, refined, or reinvented to meet local needs | |
| Trialability | The ability to test SBE on a small scale in the organization, and to be able to reverse course (undo implementation) if warranted | |
| Complexity | Perceived difficulty of SBE, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and number of steps required to implement | |
| Design quality and packaging | Perceived excellence in how SBE is bundled, presented, and assembled | |
| Cost | Costs of SBE and costs associated with implementing SBE including investment, supply, and opportunity costs | |
| Outer Setting | ||
| Student needs and resources | The extent to which student needs, as well as barriers and facilitators to meet those needs, are accurately known and prioritized by the instructor/department/institution | |
| Cosmopolitanism | The degree to which an instructor/department/institution is networked with other external organizations | |
| Peer pressure | Mimetic or competitive pressure to implement SBE, typically because most or other key peers or competing instructors/departments/institutions have already implemented it or are in a bid for a competitive edge | |
| External policies and incentives | A broad construct that includes external strategies to spread teaching strategies, including policy and regulations (governmental or other central entity), external mandates, recommendations and guidelines, pay-for-performance, collaboratives, and public or benchmark reporting | |
| Inner Setting | ||
| Structural characteristics | The social architecture, age, maturity, and size of the local context | |
| Networks and communication | The nature and quality of webs of social networks and the nature and quality of formal and informal communications within the local context | |
| Culture | Norms, values, and basic assumptions of the local context | |
| Implementation climate Tension for change Compatibility Relative priority Incentives and rewards Goals and feedback Learning climate | The absorptive capacity for change, shared receptivity of involved individuals to SBE, and the extent to which use of SBE will be rewarded, supported, and expected within the local context | |
| Readiness for implementation Leadership engagement Available resources Access to knowledge | Tangible and immediate indicators of organizational commitment to its decision to implement SBE | |
| Characteristics of Individuals | ||
| Knowledge/beliefs about SBE | Individuals’ attitudes toward and value placed on SBE as well as familiarity with facts, truths, and principles related to SBE | |
| Individual stage of change | Characterization of the phase an individual is in, as he or she progresses toward skilled, enthusiastic, and sustained use of SBE | |
| Self-efficacy | Individuals’ belief in their own capabilities to execute courses of action to achieve implementation goals | |
| Individual identification with the organization | A broad construct related to how individuals perceive the organization, and their relationship and degree of commitment to that organization | |
| Other attributes | A broad construct to include other personal traits such as tolerance of ambiguity, intellectual ability, motivation, values, competence, and capacity | |
| Process | ||
| Engaging opinion leaders Formal internal implementation leaders Champions External change agents | Attracting and involving appropriate individuals in the implementation and use of SBE through a combined strategy of social marketing, education, role modeling, training, and other similar activities | |
| Planning | The degree to which a scheme or method of behavior and tasks for implementing SBE are developed in advance, and the quality of those schemes or methods | |
| Executing | Carrying out or accomplishing the implementation according to plan | |
| Reflecting and evaluating | Quantitative and qualitative feedback about the progress and quality of implementation accompanied with regular personal and team debriefing about progress and experience | |
Implementation Research Logic Model (IRLM)
IRLM — Implementation Strategies Worksheet
The charts below contain a multitude of implementation strategies for evidence-based teaching strategies.
Several taxonomies/lists of strategies exist in the literature, including by Bunger et al. (2017) and by Powell et al. (2015). For the purposes of the workshop, choose 1 or 2 strategies from the tables below. All strategies are included as information, not to imply you must engage with all strategies at one time. For each strategy category, identify discrete strategies and operationalize them for implementing SBE.
Optional/Additional Resources
- For help selecting strategies based on your determinants of implementation, you may use the CFIR-ERIC Matching Tool found at https://cfirguide.org/choosing-strategies/.
- A full list of Bunger et al. strategies can be found at https://link.springer.com/article/10.1186/s12961-017-0175-y.
- A full list of the Powell et al. (a.k.a. ERIC) strategies can be found at https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0295-0/tables/1.
| √ | Strategy category | Example discrete strategies |
|---|---|---|
| Bunger et al., 2017; Powell et al., 2012 | ||
| Planning |
Tailor strategies
Identify and prepare champions Develop blueprint Build buy-in Assess readiness, identify barriers |
|
| Education |
Inform local opinion leaders
Conduct educational meetings Distribute materials Conduct ongoing training |
|
| Finance |
Fund/contract
Access new funding |
|
| Restructure |
Change records systems
Change structure/equipment Revise roles |
|
| Quality management |
Develop systems
Use data experts Provide supervision Give reminders Obtain student outcomes |
|
| Policy | Change policy context | |
| Expert Recommendations for Implementing Change (ERIC; Powell et al., 2015; Waltz et al., 2015) | ||
| Use evaluative and iterative strategies |
Assess for readiness and identify barriers and facilitators
Audit and provide feedback Develop and implement tools for quality monitoring Conduct local need assessment |
|
| Provide interactive assistance |
Use instructor and student feedback for facilitation
Provide local technical assistance Provide teaching supervision Centralize technical assistance |
|
| Adapt and tailor to context |
Tailor strategies
Promote adaptability Use data experts Use data warehousing techniques |
|
| Develop stakeholder interrelationships |
Identify and prepare champions
Organize instructor implementation team meetings Recruit, designate, and train for leadership Inform local opinion leaders Build a coalition Obtain formal commitments |
|
| Train and educate stakeholders |
Conduct ongoing training
Provide ongoing consultation Develop educational materials Distribute educational materials Use train-the-trainer strategies Create a learning collaborative |
|
| Support instructors |
Facilitate relay of student data to instructors
Remind instructors Develop resource sharing agreements Revise professional roles Create new instructional teams |
|
| Engage students |
Involve students
Intervene with students to enhance uptake and adherence Prepare students to be active participants Increase demand Use mass media |
|
| Utilize financial strategies |
Fund and contract for the teaching innovation
Access new funding Alter incentive/allowance structures |
|
| Change infrastructure |
Mandate change
Change record systems Change physical structure and equipment Change service sites |
REFERENCES
Bunger, A. C., B. J. Powell, K. Turner, A. S. Clary, S. L. Klaman, Y. Yu, D. J. Whitaker, S. R. Self, W. L. Rostad, J. R. Shanley Chatham, M. A. Kirk, C. M. Shea, E. Haines, and B. J. Weiner. 2017. Organizational theory for dissemination and implementation research. Implementation Science 12(62). https://doi.org/10.1186/s13012-017-0592-x.
Powell, B. J., T. J. Waltz, M. J. Chinman, L. J. Damschroder, J. L. Smith, M. M. Matthieu, E. K. Proctor, and J. E. Kirchner. 2015. A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science 10(21). https://doi.org/10.1186/s13012-015-0209-1.
Smith, J. D., D. H. Li, and M. R. Rafferty. 2020. The Implementation Research Logic Model: A method for planning, executing, reporting, and synthesizing implementation projects. Implementation Science 15(1):84. https://doi.org/10.1186/s13012-020-01041-8.
Waltz, T. J., B. J. Powell, M. M. Matthieu, L. J. Damschroder, M. J. Chinman, J. L. Smith, E. K. Proctor, and J. E. Kirchner. 2015. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the Expert Recommendations for Implementing Change (ERIC) study. Implementation Science 10(109). https://doi.org/10.1186/s13012-015-0295-0.
