3
Safety in Context

Ben Shneiderman and Mona Sloane, co-chairs of the planning committee, opened the June 26, 2024, session. Tara Behrend, Michigan State University and planning committee member, gave the keynote address, describing considerations that inspired the third installment of the workshop series.

Behrend illustrated risk management as the act of imagining possible futures. Behrend raised factors such as organizational context, incentives, dynamics, and social structures as key mechanisms in the obfuscation of anticipated outcomes of AI. The advent of high-frequency trading, a form of algorithmic financial trading, served as an example of how organizational factors color the impact of innovations. Behrend stated that high-frequency trading led to high-performance goals and pressures that nearly collapsed the financial market. Behrend offered an example of societal factors causing unintended negative impacts by describing the context and result of an AI-based surgical training. According to Behrend, students engaging in the training feared making mistakes that would be recorded by the training system. This fear caused them to perform unnaturally, impeding the effectiveness of the training. Behrend argued that understanding societal context, particularly incentives behind intended behavior, is essential to shaping policy. Domain-specific expertise is required to identify and mitigate incentives that would lead to unintended outcomes.

Behrend referenced a study that observed that team members’ ability to collaborate deteriorated after adopting Alexa, an AI-enabled voice assistant, into their workflow. This work, according to Behrend, showed that an individual’s relationship with work and coworkers changes over time when AI is introduced. Lack of knowledge regarding how AI-enabled virtual assistants and other tools might impact teams makes deploying

AI in organizational settings inherently risky, argued Behrend. Behrend listed management, applied psychology, sociology, and occupational health as key fields that can help advance the creation of resilient and healthy human–AI systems, where humans are ready and motivated to act when needed.

AI tools can also be applied to collect data rather than to enhance a workflow directly. However, tools designed to surveil workers can incentivize workers to behave abnormally, which would result in inaccurate data, according to Behrend. Behrend concluded her keynote with the hope that the workshop’s interdisciplinary discussions shed light on how to mitigate these concerns.

PANEL 1: MATURING AI AND ITS CONSEQUENCES

The first panel, designed to focus on the everyday use of AI in different organizational and industrial contexts, was moderated by planning committee member Madeleine Clare Elish, Google Cloud. The panel aimed to highlight how the use of digital tools shifts over time and adds a dimension to the concept of predicting and accounting for risk. To open the conversation, Elish asked panelists to speak about issues or topics they feel should be elevated in policy discussions around AI risk and AI safety.

Hatim Rahman, Northwestern University, noted the global nature of organizations as an impactful consideration. As regulations vary by location, organizations might need to approach enforcing safety and addressing risk differently across global offices. Employees across offices may have different concepts of what is appropriate, according to Rahman, or different levels of comfort around the use of and experimentation with AI systems.

Aiha Nguyen, Data & Society, stated that safety and risk are not shared equally. Risk, according to Nguyen, is tied to power, while safety is a value to which all individuals should have access. Power can determine how much risk is acceptable and how it is allocated. Nguyen noticed a tendency for conversations to predetermine risk as both inevitable and mitigable. It is important, Nguyen argued, to consider how risk can be shifted to minimize impact on groups that are already overburdened. Nguyen stated that governance agencies adopting AI tools play a unique role; they are in the position to make immediate changes in standards based on the risks that they surface.

Missy Cummings, George Mason University, defined safety as the freedom from risk. While risk is defined and measured, safety is a social construct of agreed levels of acceptable risk. Cummings underscored AI maintenance as a major risk that is often overlooked. AI must be updated continuously and maintained; new data must be cleaned and appropriately organized. Cummings linked this concept to the repurposing

of AI tools, particularly in safety-critical systems. Using a recent issue with Cruise self-driving cars expanding to San Francisco without sufficient training on articulated buses as an example, she argued that a system developed for use in one context should not be assumed to work in another.¹ Different contexts require additional data to understand risks and safety.

Panelists, prompted by Cummings’s remarks, discussed the impact of data on safety. Rahman noted that some consumer-facing organizations make consequential decisions related to housing, rental, credit, and bank loan approvals with insufficient data. Data sets might reflect a small population, leading to inadequate training of AI systems that are then tasked with predicting risks. Lack of data set maintenance and assumptions regarding the generalizability of data sets lead to consequential decisions that can dramatically reduce the safety of those impacted. Nguyen highlighted that AI applications can be designed to streamline support for those who need services with automated approval rather than to deny or reject services. Such an approach could reduce the complexity of data sets and minimize negative impacts.

Nguyen followed with an example of how misinformation in data sets also causes harm. New York City deployed an AI chatbot intended to help businesses understand laws that they were expected to uphold. However, according to an article from The Markup, the chatbot dispensed incorrect information that put employees and renters at risk of having their rights violated.² She argued that the output of incorrect information is particularly harmful because those who the chatbot harmed would not be automatically protected by the law. Rather, according to Nguyen, impacted citizens would need to know which laws were being violated and file a complaint to be able to protect themselves.

Panelists discussed potential approaches to mitigating unintended consequences of AI integration. Cummings argued for a chief AI officer role in all organizations that develop or integrate AI that could cause harm. She stated that this role would designate a leader to understand the technology, associated risks, and additional expertise needed to solve issues that may arise. Rahman suggested making measured progress by studying how employees engage with AI tools and making long-term decisions accordingly.

Nguyen cautioned against the assumption that AI adoption, even when done well, is always beneficial. She highlighted cases in which nontechnical solutions can solve problems and suggested that organizations carefully consider these solutions along with AI.

Prompted by Elish, the panelists discussed independent oversight strategies. Cummings argued that AI applications that can result in human injury or death should require oversight by a government entity. Nguyen agreed, adding that oversight

___________________

¹ B. Templeton, 2023, “GM’s Cruise Robotaxi vs Bus Crash Caused by Confusion Over Articulated Bus; They Say It’s Fixed,” Forbes, April 8.

² C. Lecher, 2024, “NYC’s AI Chatbot Tells Businesses to Break the Law,” The Markup, March 29.

should consider and account for how technological innovations might perpetuate existing harmful social systems. Rahman noted the use of third-party accountability systems, such as ratings and ranking systems, to encourage more sustainable practices. He expressed hope for multipronged approaches to safety in the AI space.

Panelists turned to a discussion of the impact of time on the consequences of AI system integration. Rahman raised concerns regarding the social implications of interacting with AI systems over time. Individual responses to routine use of AI systems, Rahman observed, include a reduction in human interaction and overreliance on AI outputs. Coupling these individual responses can result in a breakdown of decision-making oversight and domain-knowledge transfer. Rahman suggested organizations consider current norms around safety, accountability, and transparency prior to AI system integration. As human–AI interaction increases, organizations will need to preserve these social systems intentionally, if such preservation is desired.

Nguyen illustrated a risk case in which digital surveillance of and data-driven conclusions drawn about care workers led to a reduction of services over time.³ Extended, repressive digital surveillance of workers, Nguyen observed, has resulted in wage theft and service cuts as well as workers leaving the industry. To design tools that instead enhance worker efficiency, Nguyen stated, organizations should consider workers’ rights organizations, professional organizations, and unions, which provide a clearer vision of worker needs as well as consequences of digital investments.

Elish called attention to the popular phrase “human in the loop.” Elish asked the panelists to expand on the challenges that impact the phrase. To emphasize the need to better define the phrase, Cummings described a case in which humans were able to oversee autonomous vehicles but were unable to intervene to prevent a crash.⁴ Cummings discussed her method of familiarizing her own students with generative AI by allowing them to use LLMs for homework assignments but penalizing work with incorrect citations or misinformation. Through this, she hopes to build awareness for the strengths and weaknesses of generative AI, encouraging more mindful oversight and reduced reliance.

Elish tasked the panelists with proposing priorities for policy makers focused on AI safety and AI risk issues. Rahman endorsed prioritizing sustainability considerations such as the resources, waste, water consumption, and emissions related to AI. Nguyen suggested that policy makers could play a larger role in modeling where AI might provide the most value to society. Cummings emphasized the importance of gathering data to

___________________

³ A. Mateescu, 2021, “Electronic Visit Verification: The Weight of Surveillance and the Fracturing of Care,” Data & Society, https://datasociety.net/library/electronic-visit-verification-the-weight-of-surveillance-and-the-fracturing-of-care.

⁴ T. Mickle, C. Metz, and Y. Lu, 2023, “GM’s Cruise Moved Fast in the Driverless Race. It Got Ugly,” New York Times, November 3, https://www.nytimes.com/2023/11/03/technology/cruise-general-motors-self-driving-cars.html.

understand what is needed and what unsafe behavior means, to make evidence-based decisions regarding AI. Cummings noticed increased discussion regarding the rising national security risks of widespread AI integration. Some, Cummings noted, highlight external actors, such as Chinese self-driving cars, as the source of new risk. Rahman stated that many organizations lack the resources to build out AI systems in-house and will need to procure tools. He added that third-party, or procured, AI systems introduce additional security risks. Cummings argued that as U.S. organizations collect data, legislation preventing organizations from selling these data abroad or engaging in alternative activities that threaten users’ rights will be critical.

Rahman expressed appreciation for policy makers who work in conjunction with experts and organizations to set standards and goals to prioritize safety. Cummings added that organizations like NIST that are heavily tasked with AI management within the United States require more support through funding and cross-agency collaborations.

PANEL 2: AI INTEGRATION AND SAFETY

Ravi Parikh, University of Pennsylvania and planning committee member, opened the second panel and served as the moderator. The panel was designed to examine safety and mitigation practices among representatives from industries that have relatively mature risk management frameworks. Parikh started the conversation by asking panelists to discuss the risk management frameworks currently used in their organizations.

David Bates, Harvard University, discussed medical risk management frameworks. He pointed to a recent study that showed that one in four patients were harmed in the inpatient setting, indicating that safety methods still have room to grow in the medical field.⁵ For example, there are existing safety frameworks for subjects such as cultural awareness between medical professionals and patients. Other subjects of risk mitigation often depend on spontaneous reporting, resulting in less harm reduction than desired. Bates expressed optimism for automated harm detection, which might in the future unearth safety concerns that currently struggle to be acknowledged.

Mark Peters, Battelle, stated that his organization uses rigorous risk management frameworks for industrial, nuclear, and biological safety. Peters emphasized that in a field with a long history of risk management, safety is integrated into the culture and evident in how research and regulation are conducted. Peters suggested adapting AI risk mitigation strategies from established approaches such as nuclear safety frameworks.

___________________

⁵ D.W. Bates, D.M. Levine, H. Salmasian, et al., 2023, “The Safety of Inpatient Health Care,” New England Journal of Medicine 388(2):142–153.

Cristina Banks, University of California, Berkeley, gave a brief history of employee safety governance. The Occupational Safety and Health Act of 1970, according to Banks, gave rise to employee safety oversight organizations. In 2011, the National Institute for Occupational Safety and Health created a program called Total Worker Health to protect workers from harm and promote healthy habits. The job demand resource model and the job demand control model are examples of safety frameworks within the field of occupational psychology that identify risk factors related to demand, resources, and control. Banks mentioned ongoing efforts to identify positive methods that offer workers more resources and control, such as the freedom and encouragement to speak and act in ways that are productive for the organization without fear of retaliation. Banks suggested that approaches to AI risk management start with human-centered concerns and utilize interdisciplinary expertise to develop solutions and incentives for success.

Parikh called on the panelists to discuss existing risks in their field that make AI risk management frameworks necessary. Banks noticed that workplaces often try to change or analyze aspects of workers’ behavior. What employees want, said Banks, is sometimes incompatible with what employers want. For example, AI and other forms of automation-enabled control often conflict with the interests of the employees, as they have less power. From Banks’s understanding, AI introduced to control cost of labor has downstream effects on employee motivation, job security, and psychological safety. She detailed a case study in which a call center automated low-intensity calls to support efficiency. This change increased stress and anxiety because without the break of low-intensity calls, workers continuously completed only high-intensity tasks. Discomfort with misaligned AI tools, stated Banks, can cause employees to disengage, undermine the organization, or leave. Banks noted that involving workers early in the design and development of AI systems allows for systems to cater to both the employee and employer.

Bates echoed Banks’s concern, highlighting that frontline providers’ acceptance of AI is needed to glean benefits from the technology. According to Bates, experts are concerned that LLMs will produce medical hallucinations, which introduces the chance for malpractice but also raises legal concerns over who should be held accountable. Safety after AI integration will depend heavily on practitioners’ familiarity and experience with AI. Bates argued that humans in collaboration with AI, rather than fully autonomous systems, can allow for higher levels of performance improvements. Bates expressed concern that humans will oversee AI less closely as its accuracy improves. This too can introduce more risk into the system.

Peters addressed the global risks introduced by AI and LLMs. Peters stated that adversaries of the United States have equal access to these technologies. Peters expressed concern about the security of models, as compromised inputs can call the reliability of

information into question. Those interested in national security, according to Peters, must work to understand what technologies nations and organizations hold that could impact U.S. security and progress.

The panelists discussed how their current frameworks are evolving. Peters, Banks, and Bates expressed that they are in the earliest stages of integrating AI into their larger frameworks. Peters described interest throughout the national laboratories to put together overarching policies and governance structures regarding AI. Bates described AI committees established in the medical field to review related issues. Bates noted that organizations may be looking for AI risk management suggestions that are immediately implementable and described methods for monitoring bias as a particular area of interest.

Banks highlighted the need to understand incentives for intended behavior to inform meaningful frameworks. Banks illustrated, for example, that a system set up to reward people for being accident-free can unintentionally incentivize nonreporting. Banks described an organization replacing ineffective pre-meetings with one person collecting worker sentiment data and using ML to identify gaps, regularities, and commonalities. This approach considered the needs of workers, impactful applications of new technology, and incentives workers might have to engage with the tool such as removing the ineffective meeting and identifying subjects that need further discussion.

Parikh shifted the conversation to the trade-offs involved in integrating AI systems into organizations, offering that the benefits of AI can come at a cost. Peters underlined the risk of using LLMs with sensitive materials that can be stored. Bates noted that patients are currently unable to opt out of AI in their care; they are only able to be informed of its use. Banks highlighted the value of maintaining trust between employees and employers when AI tools are introduced. The ability to maintain trust relies on open communication, commitment to responsible use, and accountability.

Conducting the additional training needed for use of AI systems in a motivational manner can also aid in building trust, according to Banks. Banks listed free lunches and paid time as strong incentives for engagement with AI training. Continuing to discuss training considerations, Peters encouraged hands-on group training with projects and exercises to support employee understanding of what risks are introduced and how to manage them. Bates emphasized the importance of avoiding training that is lecture- or presentation-based as well as choosing days and environments that incentivize engagement. Banks suggested bringing in learning experts to design training around the intellectual curiosity and capability of employees.

Parikh then prompted participants to note examples of organizations that are making progress in developing their AI risk management frameworks. Peters praised the Pacific Northwest National Laboratory owing to its extensive experience in the use of AI and the involvement of experts in fields such as social psychology and computational

science in its process development. Bates elevated Kaiser Permanente as a leader in cataloging AI and thinking about how to monitor it, including by introducing a chief AI officer.

PANEL 3: ORGANIZATIONAL DYNAMICS AND AI

The last panel of the third workshop, moderated by Tara Behrend, aimed to elaborate on the roles of leadership and organizational culture in creating conditions that lead to safety. Behrend began by asking the panelists how they discuss AI risk and safety in their respective fields. Dorothy Carter, Michigan State University, expressed optimism for AI-enabled risk mitigation in future National Aeronautics and Space Administration (NASA) missions to the Moon and Mars. Carter also pointed to the potential benefits of AI in supporting collaboration.

Amy Edmondson, Harvard Business School, discussed the role of psychological safety in preventing accidents and failures. Discussing concerns openly and often can prevent such accidents and failures, she observed. Edmondson listed several barriers to speaking out about safety concerns: oppressive social hierarchies, fear of seeming incompetent, or fear of being seen as a negative person. Actively creating environments that support speaking honestly and candidly is necessary, according to Edmondson, to encourage a culture of safety.

Melissa Valentine, Stanford University, echoed Edmondson’s opening statement with an example of how hierarchy can have a negative impact, particularly in project development. In her example, a group of data scientists working closely with users had less power and authority within an interdisciplinary team and therefore had to compromise on data analytics practices that they found important. Valentine highlighted that the imbalance of respect for expertise is something leadership needs to monitor and address. Edmondson added that team members are often coming from different backgrounds with different ways of thinking. These differences are rarely reflected on and discussed, which leads to false assumptions regarding how individuals perceive and understand both social and technical realities (i.e., time frames, status differences, mindsets, and jargon).

Valentine noted organizational science as a relevant source of expertise for analyzing the impacts of AI systems developed or deployed in organizational settings. Valentine described a study that found that as more tasks were automated, fewer individuals had the experience to determine when and how to intervene when a problem arose. Valentine voiced concern that increased AI integration into workflows could impede the transfer of domain knowledge needed to maintain oversight.

Panelists described what leaders can do to create cultures where different people’s voices are encouraged and heard and where incentives are aligned to minimize negative consequences. Edmondson supplied that an initial step to building learning environments and environments of psychological safety is highlighting the previously discussed challenges and having an open dialogue about them. Edmondson stressed that leaders should make organizational factors and risk management routinely discussable. Leadership can make the value of people’s voices clear by encouraging questions, expressing appreciation for both positive and negative feedback, asking questions in return, and encouraging a sense of belonging. Valentine pointed to her recently published article about how to avoid ML failures by asking the right questions within the team.⁶

Behrend prompted the participants to illustrate examples of how leaders have sought critical feedback without adding fear of retaliation or have cultivated an environment where failure is seen as a learning experience. Valentine discussed a recent presentation from Lyft that she attended where a data scientist and product manager explained differences in data quality demand depending on the data set. Valentine highlighted their ability to operationalize quick audits by determining how frequently data sets needed to be reassessed, thus creating a social and technical intervention. Valentine went further to note the importance of training personnel to be able to evaluate the output of an AI system. Carter described NASA as a great example of an organization that values learning from failure and invests in building trust. Carter offered that some leaders have built a culture of trust and common perspective through human connection–oriented actions such as leaving the microphone on for astronauts on missions to hear jokes and laughter of team members on Earth.

Behrend asked the panelists about concerns regarding trust development, interdisciplinary learning, and team functioning as humans begin to interface with machines more than each other. Carter argued that AI could help optimize knowledge transfer in multiteam systems by organizing groups around who has the proper expertise to solve a given problem. Valentine highlighted that temporary interdisciplinary teams are commonly organized through individuals’ professional networks, limiting the possibility of finding experts of optimal relevant experience. Valentine noted her own ongoing work around computer-supported creation of temporary interdisciplinary teams, called “flash teams.”

___________________

⁶ D. Popovic, S. Lakhtakia, W. Landecker, and M. Valentine, 2024, “Avoid ML Failures by Asking the Right Questions,” MIT Sloan Management Review Summer.