1
INTRODUCTION

Generative AI (GenAI) is a type of artificial intelligence (AI) that produces a variety of content, including text, imagery, and audio. Large language models (LLMs) are a subset of GenAI that specializes in interpreting and generating human language to create text (see Figure 1-1). GenAI models are trained on vast amounts of text and use the content and relationships of that text to predict and generate new text (Accenture, 2024; Briganti, 2023). Users of GenAI guide the creation of text using prompts and post-processing actions to further refine and, if needed, correct potential errors, omissions, and fabrications.

GenAI tools currently have a range of potential applications in health, health care, and biomedicine. For example, GenAI can enable clinical work and reduce cognitive burden by drafting administrative documents, aiding in clinical documentation tasks, and supporting decision making (Gandhi et al., 2023). It also can help patients to better understand and engage with their health care (Pahune and Rewatkar, 2024). GenAI can also facilitate health care payment tasks by summarizing patient data and generating written authorization for health care services (Shoja et al., 2023). Clinically relevant summaries and recommendations generated by LLMs might also help provide potential diagnoses, allowing health care professionals to expedite treatment decisions for their patients (Briganti, 2023). GenAI also has the potential to assimilate scientific evidence and help identify novel therapeutics as well as prioritize potential repurposing candidates for existing drugs (Yan et al., 2024).

While these early experiences hold promise, the evidence for real-world impact of GenAI tools remains limited. Furthermore, applications of GenAI in health and biomedicine raise unique risks. These include information inaccuracy relevant to medical decision making due to so-called hallucinations or confabulations; inequitable access, utility, and applicability of LLMs in lower-resourced environments; and the perpetuation of biases present in training data or introduced by AI engineers (Yang et al., 2023; Zhang and Kamel Boulos, 2023).

As is the case with all AI tools, the safe, effective, ethical, and equitable use and implementation of GenAI in health and medicine is essential to maximizing its benefits while minimizing risks. As further expanded on below, ensuring such trustworthy and responsible use of GenAI tools in health care requires approaches that are both similar to and different than those applied to predictive and analytical AI tools. As these technologies evolve, so too must the policies and best practices that guide such trustworthy practices. Accountability mechanisms could include creating and implementing a governance framework across health systems and organizations; enhancing standards for testing and training LLMs and GenAI on diverse data sets; and requiring health care providers to have training and certification on LLMs and GenAI use in clinical settings. In addition, clinical workflows will need modification to allow for effective GenAI interaction and human oversight. Furthermore, once workflows are designed and implemented, periodic local testing, validation, and oversight of GenAI in real-world clinical settings will be essential (Ratwani et al., 2024).

Achieving the opportunities above will require many parties and skill sets to work toward the ultimate goal of upholding and promoting a continuously learning health system. A Learning Health System (LHS) has been defined by the National Academy of Medicine (NAM) as

one in which science, informatics, incentives, and culture are aligned for continuous improvement, innovation, and equity—with best practices and discovery seamlessly embedded in the delivery process, individuals and families as active participants in all elements, and new knowledge generated as an integral by-product of the delivery experience. (NAM, n.d.)

As part of a larger effort to achieve a Learning Health System and improve alignment across these domains, the NAM developed a set of Shared Commitments as a trust framework for health and health care services, emphasizing system performance that is accessible, affordable, transparent, accountable, and adaptive (McGinnis et al., 2024). To this end, the NAM Digital Health Action Collaborative convened a workshop on October 25, 2023, and a follow-on meeting with federal agency representatives on October 26, 2023, to enhance common understanding among health professionals and health system leaders, technology developers, and government agencies of the nature and health care implications of LLMs and GenAI. Workshop sessions focused on the possible GenAI benefits, risks and necessary guideposts, and guardrails in health care. This publication builds on those discussions and key takeaways, outlining GenAI’s potential opportunities, risks, and paths forward.

This page intentionally left blank.