4
Designing Paths Forward

PANEL 1: OPENING PANEL

Ben Shneiderman and Mona Sloane, co-chairs of the planning committee, moderated the first panel of the July 2, 2024, session. Panelists Jill Crisman, UL Digital Safety Research Institute, and Marc Rotenberg, Center for AI and Digital Policy (CAIDP), detailed their stances on what is needed to manage the potential risks of emerging AI tools.

Crisman began by describing the need for safety certification of AI tools by an outside organization to confirm due diligence of best safety practices. Crisman pointed to UL Solutions as an example of what an outside organization that can certify safety might look like. Crisman detailed the work that the UL Digital Safety Research Institute is currently doing to improve AI risk management, including an ongoing partnership with the Responsible AI Collaborative to build a robust AI impact database. Crisman hopes that this database will function as a history of impacts, indexed by contextual information, to inform policy and support research. Crisman expressed hope that the database will enable members of the public to report harms encountered in their daily lives. Aside from the impact database, Crisman detailed efforts to create pipelines to support organizations with the evaluation of elements such as data sets, the training of AI models, AI software systems, and deployment operations (including the release, oversight, and response to user feedback regarding AI tools). Crisman stated that oversight and accountability of the response to harms reported by affected communities could be facilitated through auditing deployment operations within an organization. Crisman

highlighted UL Digital Safety Research Institute’s work on provenance and reliable labeling of images as either “original” or “altered.”

Crisman turned to a discussion of future steps for mitigating or responding to harm. Crisman described the potential benefit of “reparations” for AI harm, provided by companies, to manage and account for risk. Crisman detailed that this practice would require risks such as death, mental and physical injury, or property and financial damage to be measured, categorized, and insured by the company that releases the AI tool. Crisman also observed that incentives such as stronger warranties might encourage incremental rather than mass deployment of new applications, affording greater opportunity for assessment and oversight.

In his opening remarks, Rotenberg noted that AI risk management conversations have primarily discussed “high” and “low” risk. Rotenberg highlighted a need to have an additional category for systems that should be prohibited. Shneiderman added that the European Union’s Artificial Intelligence Act includes “unacceptable risk” in its categorization of AI risk.¹ Rotenberg suggested that systems that cannot be controlled after they are deployed should be considered to have a prohibited level of risk. Rotenberg also commented that having AI make consequential decisions about people should be prohibited unless the reasoning for the decision can be fully explained and meaningfully contested given an adverse outcome.

Turning to a discussion of transparency, Rotenberg voiced that impact assessments should be made publicly available. To be meaningful, according to Rotenberg, assessments need to be conducted independent of the organization, prior to deployment, and during operation.

Shneiderman asked what resulted from CAIDP’s 2023 open letter to the Federal Trade Commission (FTC) calling for an investigation into OpenAI’s GPT models.² Rotenberg noted that an investigation has been opened, but a halt has not been ordered. OpenAI has already noted 11 areas of significant risk, according to Rotenberg, including cybersecurity, child safety, misinformation, and personal privacy. CAIDP filed similar complaints in the past focused on Facebook and Google, stated Rotenberg. Although Facebook was investigated, the FTC did not enforce safety measures until 8 years later. Rotenberg noticed that over that period, the Cambridge Analytica controversy and Brexit both occurred. He argued that swifter resolution from the FTC would have allowed for preventative safety measures to be put in place.

Sloane asked Crisman what lessons can be gleaned from the experience of making the impact database publicly accessible. Crisman highlighted the ability for researchers

___________________

¹ European Parliament, 2023, “EU AI Act: First Regulation on Artificial Intelligence,” https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence.

² The open letter can be found here: https://www.caidp.org/cases/openai.

to create a shared taxonomy for describing harms, incidents, and sectors. The public disclosure of the impact database facilitates a space where researchers and policy makers can come together to build broader consensus on how harms are defined and understood. Following the creation of a shared language, stated Crisman, the quantity of harms can be analyzed, and mitigation strategies can be approached. Crisman expressed interest in seeing the impact database merged with other safety databases to allow for comparisons between AI-related and non-AI-related applications and impacts.

Sloane turned to Rotenberg to inquire whether incident databases and audit trails are instructive for policy makers. Rotenberg offered that while they are necessary components of AI governance and are important to consider, they function at a more granular level than most governance frameworks. Rotenberg pointed to the Blumenthal and Hawley framework on AI legislation as an example of a framework that explicitly refers to auditing.³

Rotenberg expressed that in these early stages of regulation, CAIDP focuses on criteria for good AI governance including clear articulation of goals, independent agencies with oversight and reporting, meaningful opportunities for public comment in developing national AI strategies, and governments that integrate those comments. AI safety will require more rigorous evidence-based methods for effective assessment in the future, according to Rotenberg.

PANEL 2: WIDENING PARTICIPATION IN THE DESIGN, DEVELOPMENT, AND DEPLOYMENT OF AI TOOLS

Sheena Erete, Nathanael Fast, and Tamara Kneese, members of the planning committee, made up the second panel. They were joined by Alex Givens, Center for Democracy & Technology, as an external respondent. Erete opened the discussion by asking how participation might be translated into action. Fast noted that established metrics and classification enable the tracking of benefits and harms relating to AI tools. Givens highlighted that broader participation may yield insights that inform or drive the evolution of incentives, such as societal norms and legal accountability frameworks. Impact assessments are a potential tool to help guide the creation of safety measures. Having strong, detailed standards for responsible processes and risk disclosure that are accessible to the public who can then voice their concerns, according to Givens, allows for more consistent evaluation and accountability.

___________________

³ The framework can be found here: https://www.blumenthal.senate.gov/imo/media/doc/09072023bipartisanaiframework.pdf.

Referencing a recent report from the Center for Democracy & Technology on sociotechnical approaches to AI governance,⁴ Kneese asked what organizational resources are needed for stakeholder engagement and participatory impact assessment to be successful. Givens noted that Data & Society also released a document on this topic.⁵ Givens emphasized financial incentives, cultural norms, and regulation as key factors in both hiring the needed expertise for responsible AI and empowering employees to effect change from insights gathered during activities such as stakeholder engagement and impact assessments.

Fast encouraged further exploration of methods for expanding community feedback into all stages of AI development and use, including data collection, evaluation, and deployment. Fast noted that power can play a large role in whose feedback is considered, which subsequently influences how organizations approach community engagement. Referencing her own research, Erete stated that an intersectional analysis of power can allow for a better understanding of how to encourage meaningful engagement with community participants.⁶ Fast listed experimental methods to facilitate democratic input such as deliberative polling and citizens’ assemblies.

With respect to collaboration with local communities, Erete highlighted the need for fair compensation for participants as well as continuous maintenance of community-based relationships. Givens stated that better notice and transparency systems for those interacting with AI systems, such as employees being notified that an AI tool is evaluating them, along with user reporting would allow for community feedback. Kneese noted that slowing down the deployment of technologies can allow time for feedback to be considered before harm is caused. Givens stated that publishing the findings from and subsequent changes owing to community engagement helps to ensure that communities can see the outcomes and benefits of their work with organizations implementing AI tools.

Kneese cautioned that organizations should consider stakeholder fatigue if they are bringing in the same community experts to elicit feedback. In response, Givens highlighted the role of civil society organizations in acting as a bridge for affected communities. More work can be done, according to Givens, to support and communicate with civil society organizations who have more resources to answer questions, conduct research, and engage with communities in ways that are culturally informed. Givens

___________________

⁴ M. Bogen and A. Winecoff, 2024, “Applying Sociotechnical Approaches to AI Governance in Practice,” Center for Democracy & Technology, https://cdt.org/insights/applying-sociotechnical-approaches-to-ai-governance-in-practice.

⁵ B.J. Chen and J. Metcalf, 2024, “Explainer: A Sociotechnical Approach to AI Policy,” Data & Society, https://datasociety.net/library/a-sociotechnical-approach-to-ai-policy.

⁶ S. Erete, Y. Rankin, and J. Thomas, 2023, “A Method to the Madness: Applying an Intersectional Analysis of Structural Oppression and Power in HCI and Design,” ACM Transactions on Computer-Human Interaction 30(2):1–45.

offered that increased and reoccurring engagement with civil society organizations requires experience and understanding. Givens pointed to recent guidance from the Office of Management and Budget’s Office of Information and Regulatory Affairs on how to request input efficiently as an example of building relevant understanding.⁷

PANEL 3: EVALUATION, TESTING, AND OVERSIGHT

Abigail Jacobs, Hoda Heidari, and Tamara Kneese, members of the planning committee, made up the third panel. Erie Meyer, Consumer Financial Protection Bureau (CFPB), joined them as an external respondent. The members of the planning committee opted to engage with Meyer by asking how CFPB approaches testing and evaluation, with questions influenced by discussions from the June 20, 2024, session.

Meyer opened the conversation by discussing what CFPB has done to prepare for and address the evolving technological market. Meyer listed several teams at CFPB that work to evaluate and audit organizations in an effort to protect consumer rights. Meyer pointed to an upcoming CFPB registry that will document organizations that repeatedly violate federal orders. This registry, Meyer stated, will enable whistleblowers, states, and individuals to participate in regulation enforcement.

Meyer highlighted that clear and explicit rules allow new organizations to incorporate guidelines easily while reducing the threat of loophole exploitation from established organizations. Meyer stated that regulation established prior to the advent of AI can and should be applied to organizations even when AI systems are in use. Meyer referenced a CFPB publication that found that financial firms using chatbots could still be evaluated against existing legal obligations of explainability. This finding allowed CFPB to use standing regulations to combat AI-informed credit decisions that were unexplainable.^8,9

Kneese asked how CFPB locates and addresses harms that might not be described in technical terms by users who report an issue. Meyer discussed the publicly accessible CFPB Consumer Complaint Database, a detailed collection of user or consumer feedback, as a robust source of information on emerging harms. Qualitative data that build contextual understanding, stated Meyer, can often serve as a vector to gather and analyze quantitative data to visualize systemic issues. Meyer noted that understanding how

___________________

⁷ Office of Management and Budget Office of Information and Regulatory Affairs, 2024, “Broadening Public Engagement in the Federal Regulatory Process,” https://bidenwhitehouse.archives.gov/omb/information-regulatory-affairs/broadening-public-engagement-in-the-federal-regulatory-process.

⁸ Consumer Financial Protection Bureau, 2023, “Chatbots in Consumer Finance,” https://www.consumerfinance.gov/data-research/research-reports/chatbots-in-consumer-finance/chatbots-in-consumer-finance.

⁹ Consumer Financial Protection Bureau, 2022, “CFPB Acts to Protect the Public from Black-Box Credit Models Using Complex Algorithms,” https://www.consumerfinance.gov/about-us/newsroom/cfpb-acts-to-protect-the-public-from-black-box-credit-models-using-complex-algorithms.

users might focus their complaint allows for teams using the data to search the needed keywords to gather relevant information. In the CFPB study regarding chatbots, searching “human” in CFPB’s database surfaced data points that informed its findings, as many users expressed concerns over not being able to speak with a human rather than directly referencing the “chatbot.”

Meyer emphasized publicly accessible databases, open commission meetings, and whistleblowers as sources of information that can promote fairness. As comments from these sources can come from a variety of technical and nontechnical perspectives, Meyer encouraged regulators to hire expertise from all relevant disciplines. According to Meyer, CFPB’s investment in employing technical experts and facilitating interdisciplinary teams is a strategy that supports informed regulation. Meyer stated her interest in maintaining the quality of current data sets used by CFPB while expanding inputs of data to be more representative of all consumer experiences. Kneese stated that although the environmental impact of AI and data centers is also an important topic, it was not discussed at length during any of these workshops.¹⁰

PANEL 4: SAFETY IN CONTEXT

Tara Behrend, Madeleine Clare Elish, and Ravi Parikh, members of the planning committee, came together for the fourth panel. Diana Burley, American University, joined them as an external respondent. Parikh and Elish highlighted cases of the unintended consequences of AI integration and misalignment that were detailed in the third workshop. Burley noted that while human actions are difficult to predict, lessons learned from past integration of technologies can allow for risks and consequences to be anticipated. Elish stated that explicit definition of a given concern and detailed analysis of how AI might be leveraged to solve it should be conducted prior to the deployment of an AI tool; such discussion and investigation should be done iteratively with intended end users to ensure alignment.

Behrend asked the panel to discuss how leaders might determine when to take risks regarding AI integration and when to be cautious. Burley stated that every organization should have preexisting boundaries for acceptable levels of risk that should be applied similarly to AI tools. Leaders should, according to Burley, consider the organizational context and culture prior to tool integration to make informed decisions on when and how to introduce AI. Elish highlighted that responsible and effective integration of AI

___________________

¹⁰ The National Academies of Sciences, Engineering, and Medicine engaged with this topic during a separate workshop event: https://www.nationalacademies.org/event/43750_11-2024_implications-of-artificial-intelligence-related-data-center-electricity-use-and-emissions-a-workshop.

tools takes time and resources, considering the short- and long-term impacts of introducing new technologies.

Parikh emphasized that the function and application of new tools should be transparent to all levels of management to keep employees informed and engaged. He noted a hesitation to adopt AI, particularly in the healthcare industry, that could be overcome through increased dissemination of safety information, accessible and incentivized AI safety reporting, and training grounded in proven use cases.

Burley stated that many individuals are coming into organizations with prior knowledge of technology, which affects leadership’s ability to control how technologies are used. Parikh argued that off-label use of tools, although potentially leading to harm, is common and can be beneficial to an organization’s better understanding a tool’s benefit. Elish cautioned that users may not have enough information to extrapolate appropriate alternative applications of AI tools.

To make meaningful control possible, Elish called for more research to understand the different humans and decision-making loops involved in an AI system. Parikh offered that we might consider AI as a tool to increase efficiency or redirect decisions in a human-designed operation. Parikh reflected on Rotenberg’s call for AI to be controlled by humans at all times, suggesting instead that autonomous systems can have significant benefits if they are properly tailored for their intended purpose and their design is informed by experts.

Behrend turned to the general public’s concern regarding control of their personal data. Parikh resurfaced comments from the June 26, 2024, workshop, highlighting the limitations of the consent paradigm and difficulties consumers face in opting out of interacting with AI systems. Burley stated a need to balance protection, individual agency, and broad representation relating to data to develop high-quality models that respect individual rights. Elish and Behrend called for intentional approaches to compensating and protecting the privacy of those whose data are collected.

PANEL 5: REFLECTING ON THE NIST AI RMF

Elham Tabassi, NIST, described the intentions behind the NIST AI RMF. The NIST AI RMF begins by defining risk as the likelihood and consequence of an event. According to Tabassi, the consequence of an event can be positive, negative, or neutral. Tabassi emphasized that NIST is focused on the science of evaluation, providing the foundation for evidence-based and interoperable AI evaluations. As seen in the NIST AI RMF, this includes considerations such as investing in external independent oversight and using quantitative, qualitative, or mixed methods of evaluation as needed.

Shneiderman asked if future iterations of guidance would include different testing approaches, such as red teaming, usability testing, A/B testing, small public testing, and larger public testing. Tabassi mentioned that NIST’s ARIA program does describe the role of and need for model testing, red teaming, and field testing, and she expressed interest in expanding the coverage of testing in future iterations of the NIST AI RMF.

Sloane asked about lessons learned from developing the NIST AI RMF. Tabassi stated that reaching out to diverse experts as well as creating opportunities for public input were essential to the iterative development of the NIST AI RMF. Tabassi noted that observations that are currently qualitatively analyzed have the potential to evolve into metrics that can be quantitatively assessed. Tabassi stated that different forms of testing are required depending on context, type of risk, size of impact, and community that is impacted. The NIST AI RMF does not make recommendations on thresholds for safety, according to Tabassi, due to the variety of assessment and risk mitigation strategies.

The NIST AI RMF hopes to maximize the benefits of AI technology while minimizing negative consequences and harms. Tabassi stated that a risk-centered approach can unlock possible pathways for innovation by encouraging deeper investigation into the possible applications of emerging technologies. The NIST AI RMF, according to Tabassi, is meant to foster a culture of thinking about risk and impact as early as possible, not only within organizations developing and deploying AI systems but also between users.