Appendix C

Participants’ Ideas for Metrics of Engaged Research

This appendix presents the ideas that participants contributed for advancing measurement of engaged research at the project, institution, and meta-network levels. The input from in-person and virtual participants were collected using Slido, an online tool that enabled interactive polling. Participants responded to the following prompt:

If we were to measure what we value, what should we be measuring? What measures most need to be developed? Consider both project and institutional measures, as well as process and outcomes.

Participants’ responses are detailed below, organized by nine themes: relationships, trust, competence and capacity for engagement, sustainability, co-production and co-benefits, focus on communities, focus on institutions, focus on meta-networks, and other ideas and resources.¹

RELATIONSHIPS

• Relationship building (Brian Wampler and Ian Binns)
• Relationships (Jenny Irons)
• Quality of relationships (Michael Rios)
• Relationship satisfaction, quality, longevity (at project/individual level, stories and ratings from all partners) (Elyse Aurbach)

• Additional relationships enabled thanks to the co-produced research (Sonia Hall)
• Community partners’ view on the strength and value of the relationships (Sonia Hall)
• Increase in positive feelings between researchers and local communities (Lexi Shultz)
• How do we measure/evaluate how far along a project/relationship is? In other words, when is a relationship ready to apply for funding or other types of support? (Brian Wampler)
• Value of relationship to external partner vs. value of relationship to university-based team (Brian Wampler)
• Relationships in its all their various attributes (Mahmud Farooque)
• Is the partnership resilient and able to survive turnover of members? Is the work and network of relationships strong enough to survive if an influential champion leaves? (Kacy Redd)
• Relationships and trust (Rachel Wurzman)
• Combined qualitative and quantitative metrics of relationship building and connectivity: the number of trainings and professional development not just offered to community but led by community (Eboné Lockett)
• Strength of engagement as research impact (Benjamin Olneck-Brown)
• Activities conducted, number of engagements, formal agreements, number of community members reached, number of community members trained, social outcomes, new resources developed, acceptability of interventions implemented, policy changes, funding distributed to partners, perceptions of co-design and collaboration (Tamara Haegerich)

TRUST

• Trustworthiness (Mary Jo Callan and Faith Uwadiae)
• Measure qualities of a “trustworthy process” (Tim Steffensmeier)
• We should start by asking our partners these questions and being ready to take the time to repair harm, build trust, and establish relationships before asking this question. Depending on the stage of the partnership and past experiences, it is likely that past harm and distrust will result in answers to this question that may not truly reflect the partner’s truth—they may just be telling you what they think you want to hear: for example, schools defaulting to “student achievement” because that is what they are told by their state. (Kim Wright)
• I really like the “trustworthiness” reframe. I would add that one concept in learning sciences that is important is politicized trust,

• which acknowledges the location of trust building within networks of power. (Bill Penuel)
• People feel safe in a process as a measure of collaboration (Adam Parris)
• Trust for research can come from support for science (and understanding of science) by religious communities. The putative “conflict” between religion and science needs to be addressed. (John L. Burch)
• Trust and community empowerment: awareness, agency, action (Marisol Morales)
• How many activities are initiated by the community? (Marisol Morales)
• How does higher ed institution become a preferential partner because trust exists? (Marisol Morales)
• Measure depth and pervasiveness of partnership (Marisol Morales)
• Measure preparation of students and university-affiliated folks to enter community respectfully and humbly (Marisol Morales)
• As a citizen myself, in an effort to better set diverse, equitable, inclusive, and accessible targets for allocating our limited resources, I feel improvements could be made to assess the presence of secure spaces to have vulnerable conversations about community needs and values representative of local demographics and geographies (Charles Smeltzer)
• Trustworthiness and systems engagement with partners toward actionable solutions and change (Nadine Barrett)

COMPETENCE AND CAPACITY FOR ENGAGEMENT

• Engagement competencies (Elise Cappella)
• Engagement supports (trainings, mentoring, incentives, coaching, recognition) (Elise Cappella)
• Cultural humility (Michael Rios)
• We need to determine what qualities are really required to do community-engaged research, such as trustworthiness, and then ensure that funders, professors, and universities are really looking for and assessing the level of this quality when anyone from their organization (or someone they are going to fund, in the funder’s case) is doing work with communities (Katie)

SUSTAINABILITY

• Institutional track record for delivering on sustainability in a variety of ways, including internal sustaining, partner sustaining, embedding, spinoff into new independent organizations (Jessica Bennett)

• Is the intervention attracting sustainable/ongoing funding? (Kacy Redd)
• Retention and advancement of engaged researchers throughout the academic career path: undergrad through full professor (Erhardt Graeff)
• Map where possible to U.N. Sustainable Development Goals, including good health and well-being, reducing inequalities, and partnerships (Arthur Ellis)
• Sustainability of partnerships to maximize an approach that starts with partnership and not the specific funding opportunity. We often blame funders for making us approach community partners at the 11th hour or after an investigation is fully planned. If we focus instead on cultivating enduring partnerships, research projects, questions, and funding proposals will flow from those. (Mary Jo Callan)
• Policy change; community ownership and sustainability of intervention and programs: measures have to be tailored to what the program was designed to affect, which is hard to aggregate (Laurie Van Egeren)

CO-PRODUCTION AND CO-BENEFITS

• Specific changes that the co-produced research informed, influenced, or amplified (Sonia Hall)
• For all outcome-based measures: is there alignment in view of progress and impact among all partners? (Sonia Hall)
• Decisions that were changed in response to the co-produced research results and in response to the co-production process (Sonia Hall)
• Co-develop measures of the outcomes that community members care about and outcomes they can use for advocacy beyond the research project (Heidi Schweingruber)
• Measures of partnership quality (Laurie Van Egeren)
• Change and evolution in research questions and approaches through the co-production process (Sonia Hall)
• The co-creation of research questions to address, as well as methodologies: that is, the development of a collaborative research agenda is an outcome in itself (Lexi Shultz)
• Increased institutional recognition of and investment in co-production and community science (Lexi Shultz)
• Increased participation and sense of co-ownership (Nova)
• Numbers and percentages of research project presentations at academic and public events that are led or co-presented by community partners (with funding for their travel and time included in grants) (Erhardt Graeff)

• Partnership equity and effectiveness, including in communication, resources, decision making, engagement, and influence (Elise Cappella)
• Perceptions and experience related to equity and benefit of a partnership (Mary Jo Callan)
• Measurement of how accurately partners feel that their experiences and expertise were captured and represented by researchers (Deondra Rose)
• We should measure experiential outcomes related to the processes engaged in, for both researchers and community members. Measures that can influence change of practice and research are important. (Douglas A. Watkins)
• Components of the participatory process and measure each of these (Susanna Campbell)
• Engaging the community in identifying the problem, developing the measures, and developing the solutions (Nadine Barrett)
• Equity in partnerships and collaborations (Nadine Barrett)
• Actual systems change informed and co-developed with community expertise (Nadine Barrett)
• Level of agreement that a collaborative endeavor empowered participants to take action (Adam Parris)
• For all outcome-based measures, is there alignment in view of progress and impact among all partners? (Sonia Hall)

FOCUS ON COMMUNITIES

• Whether the community thinks the problems they prioritize are being addressed (Heidi Schweingruber)
• Community capacity building (Michael Rios)
• Needs of the community and barriers to meeting those needs (Jo)
• For a specific focus—water quality, for example—the community sees improvement or has specific data they can use to push for improvements (Lexi Shultz)
• Measures of whether and how the university is providing learning opportunities for researchers to develop capacities to work with communities (Heidi Schweingruber)
• Implementation measures, such as RE-AIM,² with a focus on community populations so we start getting at sustainability of interventions in communities (Jeanette Waxmonsky)
• At institutional level, where community partnerships are happening, how they are brokered, who is involved (and their satisfaction), relationship with discipline, etc. (Elyse Aurbach)

• Level and diversity of engagement by community partners, as a proxy for whether they see the value in the partnership (Sonia Hall)
• Agency of community partners in affecting change (Sonia Hall)
• Subjective quality of life of the community: to analyze “soft” measures, well-being, feelings of people, including qualitative measures (Gerlinde Kristahn)
• Measures impacts that undergird economic development/prosperity (Tim Steffensmeier)
• First need to capture what communities see of value in a partnership with research institutions/researchers. Measure whether those needs/objectives are being met. They don’t always want data and research. (Elsa Falkenburger)
• Outputs and products for the community, and whether they are used and applied by the community in pursuit of their goals (Sonia Hall)
• Measure the stabilizers created with community input. How effective were they in instilling ownership and stability? (John James)
• Improved quality of life, sustainability, empowerment of communities (Nova)
• We need to move beyond activity to actual impact. Numbers are important, but we need to know if what we are doing actually makes a difference in people’s lives. (Susan Renoe)
• Impact on public health outcomes, impact on democratic agency: Do people feel more empowered to participate in democracy more broadly as a result of participating in engaged research? Creation and value of new collaborative relationships (Adam Levine)
• Community and collaborator perceptions of whether and how the research effort addressed their needs (Kaytee Canfield)
• Change in quality of actions taken or decisions made, using metrics the user organization employs (Lawrence Friedl)

FOCUS ON INSTITUTIONS

• Measures and benchmarks for institutional change: What does an engaged university look like in this moment? What should be seen as foundational expectations vs. more contextually situated? (Benjamin Olneck-Brown)
• The proportion of engaged research projects at our institution that were catalyzed by a community-identified agenda or use-case in comparison to the number that come from an institutionally identified agenda or use-case (Lina Dostilio)
• Is the partnership being shared with decision makers—including state legislatures, community leaders, institutional president,

• donors, alumni, or industry leaders—in press releases, briefings, or other dissemination venues? (Kacy Redd)
• Understanding how to align individual perceptions of individuals in the institution to the institutional level (Prajakta Adsul)
• How do we capture if a community-based activity promotes the values of the institution? (Brian Wampler)
• Policy and infrastructure support: evaluate institutional changes that support community-engaged research, such as the creation of offices, staffing, and funding dedicated to such initiatives. This might include tracking changes in policy that facilitate community access to university resources. (Richard A. Tankersley)
• Long-term engagement: measure the sustainability of community partnerships beyond individual projects. Metrics could include the number of multiyear community projects supported by the institution or the renewal rates of community partnership agreements. (Richard A. Tankersley)
• Cultural competence: monitor the institution’s progress in fostering an environment that respects and incorporates diverse cultural perspectives. This could be measured through diversity and inclusion training participation rates and feedback on institutional climate from community partners. (Richard A. Tankersley)
• Number of academic institutions offering public-facing reports specifically touting engaged scholarship and community partnerships (Erhardt Graeff)
• Outlets for researchers other than publications (Lexi Shultz)
• Measure how institutional resources—such as dollars, in kind contributions, power—are accessible or being leveraged for community priorities and community benefit (Elsa Falkenburger)
• Tracking university investments in support for community-engaged research—such as payments to community members, training of researchers, and staff to support partnership building (Heidi Schweingruber)
• Percentage of grants awarded that offer significant time and resources for relationship building and compensating non-academic partners (Erhardt Graeff)
• Also need to better capture context and contextual conditions within an organization (Prajakta Adsul)
• Institutional culture and support for this and other work (e.g., DEI [diversity, equity, and inclusion]) via surveys, focus groups, etc. (Elyse Aurbach)
• Revision of P&T [promotion and tenure] to reflect the kind of work needed for community engagement and the time it takes to do it well (Heidi Schweingruber)

FOCUS ON META-NETWORKS

• We need measures that are accessible to researchers, practitioners, and partners. This includes evaluations and assessment measures, both qualitative and quantitative. These can be held at the national or institutional level so there is consistency in utilizing promising practices. (Susan Renoe)
• Many measures already exist but we aren’t aware of across disciplines: How can our meta-network support cross-fertilization on the measurement and adaptation side? (Emily Ozer)
• Review and assess portfolio and quality of research and community partnerships. Measure number of grants and programs funded with community partnerships. Monitor across the depth of community partnerships on a continuum from being involved as a part of an advisory board or participating in a focus group to serving as a PI [principal investigator] or co-PI. (Nadine Barrett)

OTHER IDEAS AND RESOURCES

• The research in question moves from usable to used; there is a documentable approach and learning for uptake of usable data. (Lexi Shultz)
• Framework for evaluation of RPPs [requests for preliminary proposals] that we’ve iterated since 2017 and developed both associated measures and sense-making routines for them: five dimensions relate to trust, supporting partner organizations in meeting their outcomes, co-involvement in research, producing original research, and building capacity for ongoing joint work³ (Bill Penuel)
• Increased participation in STEM [science, technology, engineering, and mathematics]-identifying individuals (Lexi Shultz)
• We look at epistemic justice by gathering data that are easily collected in the end of class about students’ experience of the classroom and use them systemically to try and address the education debt owed to racially minoritized students, and also girls and gender nonbinary students.⁴ (Bill Penuel)
• Engage evaluators throughout project life-cycles to assess and substantiate impacts in social and economic terms (Lawrence Friedl)