___________________

¹ This workshop was organized by an independent planning committee whose role was limited to identification of topics and speakers. This Proceedings of a Workshop was prepared by the rapporteurs as a factual summary of the presentations and discussions that took place at the workshop. Statements, recommendations, and opinions expressed are those of individual presenters and participants and are not endorsed or verified by the National Academies of Sciences, Engineering, and Medicine, and they should not be construed as reflecting any group consensus.

field of integrated diagnostics, including the purpose, goals, and components of integrated diagnostics.” It featured presentations and panel discussions on a range of topics, including:

• current efforts to develop and implement integrated diagnostics to inform treatment decision making in cancer care;
• implications of integrated diagnostics for clinician training, care workflows, and the organization of care teams;
• opportunities for evidence generation to inform validation, clinical utility, regulatory oversight, and insurance coverage for integrated diagnostics;
• strategies for ongoing quality assurance, evaluation, and refinement of integrated cancer diagnostics based on new evidence; and
• mechanisms to enable broad patient access to integrated diagnostics, particularly in community-based settings of cancer care.

This Proceedings of a Workshop summarizes the presentations and discussions from the workshop. Observations and suggestions from individual participants are discussed throughout the proceedings and highlighted in Boxes 1 and 2. Appendixes A and B provide the Statement of Task and agenda, respectively. Speaker presentations and the workshop webcast are archived online.²

OVERVIEW OF THE CURRENT STATUS OF AND VISION FOR INTEGRATED DIAGNOSTICS

An Evolving Integrated Data Science Approach to Personalized Oncology Care

There is currently no consensus definition of integrated diagnostics, said Kojo Elenitoba-Johnson, inaugural chair of the department of pathology and laboratory medicine and James Ewing Alumni Chair of Pathology at MSK. He referred to Hricak’s working definition as “the convergence of imaging, pathology, and laboratory testing, supplemented by advanced information technology, which has enormous potential for revolutionizing the diagnosis and therapeutic management of many diseases, including cancer.”

___________________

² See https://www.nationalacademies.org/event/03-06-2023/incorporating-integrated-diagnostics-into-precision-oncology-care-a-workshop (accessed May 26, 2023).

___________________

³ A tumor board is “a treatment planning process in which a group of cancer doctors and other health care specialists meet regularly to review and discuss new and complex cancer cases. The goal of a tumor board review is to decide as a group on the best treatment plan for a patient. These meetings can involve specialists from many areas of health care, including medical oncologists, radiation oncologists, surgeons, pathologists, radiologists, genetics experts, nurses, physical therapists, and social workers.” See https://www.cancer.gov/publications/dictionaries/cancer-terms/def/tumor-board-review (accessed December 27, 2023).

⁴ See https://www.mskcc.org/msk-impact (accessed August 30, 2023).

___________________

⁵ See https://www.oncokb.org/ (accessed August 30, 2023).

___________________

⁶ See https://commonfund.nih.gov/bridge2ai (accessed May 26, 2023).

⁷ See https://www.go-fair.org/fair-principles/ (accessed December 27, 2023).

⁸ Synoptic reporting “is a method of clinical documentation that captures and displays specific data elements in a specific format.” See https://radiopaedia.org/articles/structured-reporting (accessed December 27, 2023).

___________________

⁹ See https://www.facs.org/media/avukq4nc/coc_standards_5_3_5_6_synoptic_operative_report_requirements.pdf (accessed May 26, 2023).

___________________

¹⁰ See https://build.fhir.org/ig/HL7/genomics-reporting/ (accessed May 26, 2023).

¹¹ See http://hl7.org/fhir/us/mcode/ (accessed May 26, 2023).

¹² See https://recist.eortc.org/ (accessed August 31, 2023).

integrated diagnostics pipeline for clinical trial matching.¹³ Osterman also noted the opportunity to improve the interoperability of clinical protocols so they can readily be launched across multiple institutions, and the Clinical Trials Rapid Activation Consortium is working to make this a reality (Osterman et al., 2023).

Lawrence Shulman, professor of medicine and director of the Center for Global Cancer Medicine at the University of Pennsylvania Abramson Cancer Center, emphasized the importance of structuring patient-reported outcomes and the clinician’s overall assessment of a patient for inclusion in integrated diagnostics. Mia Levy, chief medical officer of Foundation Medicine, pointed out that it can take many years for consensus standards to be developed and then implemented by vendors, and standards-setting bodies are generally volunteer organizations. She inquired how the development of standards might be accelerated. Osterman highlighted the differences between the HL7’s approach in proposing standards for structured clinical genomics data and the approach taken by mCODE. The HL7 process was much slower because it was intended to be full scope at implementation and therefore necessarily considered all use cases during development, Osterman explained, whereas mCODE was developed based on two use cases, and it was understood from the start that it was a minimum set of common data elements that would not fully cover all use cases. He pointed out that the mCODE governance structure allows for rapid feedback from users and rapid addition of new data elements. Osterman added that mCODE follows the FHIR maturity model,¹⁴ and a committee of mCODE implementers works to ensure that new additions are compatible with current installations.

Several speakers discussed the transition from manual to automated annotation and segmentation of image-based diagnostics. Hricak described the extensive and time-consuming manual work needed to annotate images and the need to assemble very large datasets of annotated images for training AI algorithms. She emphasized the importance of collaboration among radiology, pathology, and AI experts in developing and validating these algorithms. Hricak also called for a standard lexicon to convey the degree of diagnostic certainty. Elenitoba-Johnson highlighted infrastructure challenges to achieving automated annotation, such as the U.S. health care system’s lack of any standardized mechanism to pay for the expertise and effort spent on manually annotating images at the scale required to train AI algorithms. “Somewhere in our health care delivery system, we have to account for the individuals who

___________________

¹³ See https://clinicaltrials.gov/about-site/about-ctg (accessed October 24, 2023).

¹⁴ See https://confluence.hl7.org/display/FHIR/FHIR+Maturity+Model (accessed January 25, 2024).

are going to do that work,” emphasized Elenitoba-Johnson, rather than relying on the current patchwork funding.

Levy agreed that lack of annotated images is a workflow problem, with no incentive for annotation to be completed. However, a drawback to incentivizing time spent for annotation could be that clinicians and health systems may be reluctant to lose reimbursement in favor of automated annotation. Sohrab Shah, chief of computational oncology and Nicholls-Biondi Chair of the department of epidemiology and biostatistics at MSK, noted that “manual, precise … and thorough segmentation is completely unscalable to the volume that is needed to actually train the models.” He raised the possibility that semiautomated, large-volume datasets, although potentially less accurate, could be an option for training algorithms. He wondered whether the decisions of radiologists and pathologists could be captured as they are reviewing images in their routine workflow and whether that information could be used for training. Hricak pointed to self-supervised algorithm training (discussed further below).

EFFORTS TO DEVELOP, IMPLEMENT, AND USE INTEGRATED DIAGNOSTICS

Representatives from academic medical centers, industry, and large health care organizations shared their perspectives and lessons learned from efforts to develop, implement, and use integrated diagnostics in clinical practice.

Perspectives from Academic Medical Centers

Overcoming Cultural Barriers to Change

Gabriel Krestin, emeritus professor of radiology at Erasmus Medical Center, University Medical Center (MC), Rotterdam, shared lessons learned from implementing integrated diagnostics at Erasmus MC. To provide context, Krestin explained that the Netherlands has a highly regulated national health care system. Erasmus MC is the largest academic medical center in the Netherlands, with patient care organized into seven divisions, one of which is Diagnostics and Advice, incorporating the departments of pathology, laboratory medicine, medical imaging, and pharmacy.

Krestin said that the vision for Diagnostics and Advice is to implement an integrated approach that encompassed three key steps: requests for diagnostic testing are sent to a front office; tests are performed in the appropriate department; and a comprehensive, integrated, final report is provided to the multidisciplinary care team. He said a survey found that hospital leadership and information technology (IT) vendors supported this approach, but there

was unexpected resistance from physicians. Krestin explained that pathologists expressed concerns about digitization turning pathology into a commodity and radiologists expressed concerns about additional workload and responsibilities. Referring clinicians expressed concerns about ceding control of diagnostic decision making for their patients.

Krestin explained that Erasmus MC began by implementing integrated diagnostics for several complex diseases for which the institution expected particular benefit—for patients and the health care system. However, referring physicians responded that integrated diagnostics were unnecessary, because they could consult clinical practice guidelines for complex diseases. Erasmus sought to understand how well guidelines were being followed for these types of diseases. Krestin cited a retrospective review of 604 cases in which patients had incidental adrenal findings; of these, less than 15 percent followed Erasmus guidelines that called for both imaging and biochemical workups (de Haan et al., 2019). Retrospective reviews of guideline adherence for other scenarios yielded similar results. Even with this evidence of limited guideline adherence, “we couldn’t convince the clinicians that integrated diagnostics would be the way to go forward,” said Krestin.

A second attempt at implementing integrated diagnostics was to launch a pilot study to demonstrate how integrated diagnostics could improve workflow by addressing discrepancies between radiology and pathology findings before presenting a patient’s case to the multidisciplinary tumor board. Krestin provided an example showing agreement in diagnosis between radiology and pathology in approximately 75 percent of 89 patients with suspected lung cancer. An integrated approach led to reconciliation of an additional 20 percent of patients (unpublished data). Krestin pointed out that despite these results, hesitancy about integrated diagnostics among referring physicians persisted.

Krestin underscored that the main barrier to implementing integrated diagnostics was cultural (i.e., resistance from physicians). He said the necessary elements for successful implementation include transitioning from a process of multiple sequential orders to order sets (combinations of tests that support the determination of a comprehensive final diagnosis); development of an order management system that generates the order sets and defines the materials needed for testing; and structured, integrated reports delivering the combined results. Krestin added that Erasmus MC is now looking to integrate clinical and omics data for use in predictive analytics.

Patient-Centric, Functional Alignment of Diagnostic and Administrative Components

“Integrated diagnostics is a functional alignment of the meaningful diagnostic and relevant administrative components for a specific patient,” said

Jochen Lennerz, medical director of the Center for Integrated Diagnostics (CID) and associate chief of pathology at Massachusetts General Hospital. “It is not just the integration of multiple modalities.”

Lennerz described the work of CID, which aims to bridge the gap between clinical research and clinical practice. CID integrates a research laboratory, a clinical laboratory certified under the Clinical Laboratory Improvement Amendments¹⁵ and approved by New York State, faculty in pathology, and faculty in pathology informatics. CID runs approximately 14,000 clinical tests per year, with a baseline battery of tests and an agile infrastructure that facilitates both bringing new tests on board (including adopting payer policies) and removing tests.

Achieving diagnostic quality requires aligning the diagnostic components within the health care ecosystem, Lennerz explained. He described diagnostic quality as the sum of the quality of the diagnostic test itself, plus the quality of the diagnostic procedure (i.e., the execution of the test), and the quality of the diagnostic service (Lennerz et al., 2023). The integration process for a quality diagnostic test involves assessing the technology, determining financial sustainability (i.e., reimbursement by payers), and integrating it at the relevant laboratory.

As an example of implementing integrated diagnostics in practice, Lennerz described the testing approach to determine eligibility for targeted therapy among patients with lung cancer. The process involves collection of a biopsy sample, diagnosis by frozen section and fine needle aspiration, next-generation sequencing of extracted DNA, reporting of the genotyping results to the specialty pharmacy for dispensing of medication, and initiation of targeted therapy, which he said would ideally occur within 48 hours. In the example of initiating osimertinib¹⁶ therapy for patients who have lung cancer with EGRF mutations,¹⁷ the median time from the biopsy order to initiating therapy in a rapid specialty pharmacy cohort was 5 days compared to around 40 days in a nonrapid cohort (Dagogo-Jack et al., 2023). The potential of integrated diagnostics, he said, is “to work as seamlessly as possible to achieve your intended outcome.”

___________________

¹⁵ See https://www.cms.gov/medicare/quality/clinical-laboratory-improvement-amendments (accessed January 30, 2024).

¹⁶ Osimertinib is a kinase inhibitor used to treat or inhibit the formation of non-small-cell lung cancer. See https://medlineplus.gov/druginfo/meds/a616005.html (accessed September 1, 2023).

¹⁷ EGRF stands for epidermal grown factor receptor, and a mutation in this protein can cause uncontrolled growth, leading to cancer. See https://www.lung.org/lung-health-diseases/lung-disease-lookup/lung-cancer/symptoms-diagnosis/biomarker-testing/egfr (accessed September 1, 2023).

tion to providing opportunities for precision health, these models show that multimodal data integration can also support research and development, citing two examples of studies of integrated diagnostic approaches to predicting response to therapy (Sammut et al., 2022; Vanguri et al., 2022). Salto-Tellez shared that the Royal Marsden Hospital and ICR have jointly launched the Integrated Discovery and Diagnostics initiative, which merges genomic, histologic, radiologic, and clinical data, health care economics data, and research and development data from clinical trials.

One key challenge for multimodal integration of data across clinical silos is the extent to which data need to be curated versus obtained in an automated fashion directly from the source, said Salto-Tellez. He noted that Royal Marsden Hospital and ICR approached 20 vendors to help address this challenge. Of the nine that responded, six said they could offer a holistic approach; they had higher technical capability and cost. Three vendors said they could not integrate everything but could offer a piecemeal process that could help (they had lower technical capability and cost).

Salto-Tellez also described work by a consortium of the Precision Medicine Centre of Excellence at Queen’s University Belfast, Roche, and Sonrai to improve early detection of colorectal cancer. The consortium has developed an integrated diagnostics workflow connecting the pathology and genomics workflows to support the research and development process. The informatics partner, Sonrai, has developed the data analytics algorithms that link the two workflows together at multiple junctures, such as one designed to optimize the selection of tumor material for molecular testing based on histologic staining and digital pathology and to automatically macrodissect the tumor material. Salto-Tellez noted that validation studies of the integrated algorithms are underway.

Improving Diagnostics Through Close Partnership of Radiology and Pathology

Mitchell Schnall, Eugene P. Pendergrass professor of radiology and chair of the department of radiology at the University of Pennsylvania Perelman School of Medicine, observed that diagnostics has long been part of the art of medicine, often requiring the clinician to gather information from multiple sources to reach a diagnosis. However, the exponentially expanding volume of diagnostic data makes it impractical for one individual to integrate all of these inputs. Schnall said that data science can be leveraged to integrate a patient’s diagnostic information and present it to the clinician to review and act on, similar to how the cockpit of a plane presents all the necessary information to a pilot (Schnall et al., 2023).

Schnall discussed using partnerships to address silos, emphasizing that a closer partnership is needed between the two largest diagnostic disciplines—

radiology and pathology. He described this as a “natural partnership” because they have complementary expertise and approaches and face similar clinical and operational challenges. For example, pathology brings expertise in biology, informatics, and quality systems to the partnership, Schnall explained, and radiology brings expertise in anatomy and physiology, data science and IT, and workflow.

Schnall outlined the University of Pennsylvania’s approach to establishing a closer partnership between radiology and pathology, highlighting strategies in five tactical areas:

• Joint research includes building a joint data science program focusing on image analytics and overall diagnostic analytics to improve efficiency, defining the value of diagnostic outcomes, and developing novel combined modality diagnostics.
• Harmonization of systems includes using a single medical record system, the same picture archiving and communication system, and developing diagnostic decision support.
• Education includes cross-department house staff rotations, combined modality fellowship training, and a joint informatics/innovation fellowship.
• Joint clinical activities include codevelopment of diagnostic pathways, joint diagnostic consultation services, and promoting interaction through colocation of select pathology and radiology services.
• Integrated activities include the management and communication of results; decision support; and IT, billing, and marketing activities.

Schnall also said that the University of Pennsylvania recently launched the Center for AI and Data Science for Integrated Diagnostics (AI2D). Codirected by faculty in radiology and pathology, AI2D focuses on addressing the challenges of integrated diagnostics. “Diagnostics is critical to precision medicine,” Schnall concluded, and “closer integration of radiology and pathology will improve the diagnostic process.”

Integrated Diagnostics for Earlier Detection of Cancer

Garry Gold, Stanford Medicine Professor of Radiology and Biomedical Imaging and chair of the Department of Radiology at Stanford University, discussed the wide-ranging applications of integrated diagnostic approaches to early detection of cancer, when treatment is more likely to be successful.

Gold first described several studies by the late Sam Gambhir, a founder of the Canary Center at Stanford for Cancer Early Detection and pioneer in the integration of radiation and pathology. In one example, Gambhir assessed

whether integrating FDG-PET imaging with assessment of circulating tumor cells (Nair et al., 2013) or circulating tumor microemboli (Carlsson et al., 2014)¹⁸ in patients with non-small-cell lung cancer could improve diagnostic accuracy over using radiology or pathology markers alone. Ghambir also initiated the Baseline Study,¹⁹ a longitudinal study collecting baseline clinical, imaging, molecular, and other data from 10,000 participants to characterize what “normal” values are and changes that occur in association with disease.

Gold highlighted a novel approach known as “theragnostics,” which combines diagnostic testing with therapy, such as the integration of diagnostic radiology and targeted molecular radiotherapy. The Canary Center is also developing new in vitro diagnostic (IVD) tests for early cancer detection, including the Exosome Total Isolation Chip, which can detect extracellular vesicles in clinical fluid samples (Liu et al., 2017); a magnetic wire inserted into a vein to capture circulating tumor cells that have been immunolabeled with magnetic particles (Vermesh et al., 2018); and an approach that correlates detection of volatile organic compounds in breath with PET-CT (Vermesh et al., 2022).

Gold highlighted wearable and in-home monitoring technologies as another opportunity for integrated diagnostics. One example of a wearable technology is a microneedle patch on the arm that integrates sampling and molecular testing to continuously monitor for proteins in interstitial fluid. Gold also noted that integrated diagnostics are being developed in other fields, such as combining neuroimaging and histopathology data to assess traumatic brain injury.

Achieving the Vision of Integrated Diagnostics

Several panelists discussed potential actions that could help realize the vision of integrated diagnostics in precision cancer care. “It’s early days in integrated diagnostics,” Salto-Tellez said, with a need for “significant and specific investment in this area.” Lennerz encouraged an increased focus on regulatory science and actively engaging with regulators on issues related to practical implementation (e.g., the data and performance metrics needed for radiologists to fully integrate AI) (Lennerz et al., 2022). Schnall stressed that broad collaboration among academia, industry, regulators, and payers is necessary

___________________

¹⁸ Circulating tumor cells are cells that have separated from primary or metastatic tumors. Circulating tumor microemboli are multicellular aggregates that include circulating tumor cells, and both of these can seed metastatic cancer in disparate sites in the body (Tao et al., 2022).

¹⁹ See https://medicine.stanford.edu/annual-report-2018/the-project-baseline-study.html (accessed September 1, 2023).

to achieve effective integrated diagnostics. Gold said the next generation of scientists and clinicians need to be able to communicate across disciplines and implement an integrated approach to diagnostics. “We won’t be able to truly integrate until we have people who understand the entire landscape,” he said.

Several participants discussed the challenges of integrating the vast volumes of data generated by consumer wearables and in-home monitoring technologies. Shah raised the issue of access to these data by hospital systems or clinicians, noting that wearables are often commercially developed, and the developers often have their own interests in the data collected. Gold agreed and noted the regulatory, payer, and patient privacy concerns to be addressed.

Perspectives from Industry

Opportunities to Leverage AI in Diagnosis

Dorin Comaniciu, senior vice president of Artificial Intelligence and Digital Innovation at Siemens Healthineers, discussed several opportunities to leverage AI for integration and use of the massive volumes of health data generated by medical technologies and devices. He said that computational power and bandwidth for information exchange are increasing exponentially, including low-cost supercomputing power available at the point of care.

Comaniciu said that one approach to handling the ever-increasing volume of data is AI self-supervised learning.²⁰ He cited a study of self-supervised learning using 100 million medical images, which found that it resulted in improved accuracy, more robust results, accelerated training, and improved scalability (Ghesu et al., 2022). Comaniciu noted that it would take 35 years for one person to look at each of the 100 million images for 10 seconds.

The results of self-supervised learning would need to feed into a harmonized user interface, which Comaniciu referred to as a “diagnostic cockpit,” with access to clinical information, the results of AI analyses, advanced data visualization, actionable reporting, and research and innovation. Comaniciu pointed out that structuring data has been the general approach to managing complex clinical data so it can be integrated. However, AI has the capability to shift the model from one in which the clinicians sort through structured data to one in which clinicians ask questions (comparable to Internet searching). He said large language models will support connectivity and access, provide interpretation support, and aid in synthesizing information. In addition,

___________________

²⁰ Self-supervised learning is an ML technique that teaches a model to predict hidden parts of an input using other parts of the input that are visible to the model. It can be used to perform tasks, such as image comprehension and object detection (Rani et al., 2023).

this workspace could incorporate virtual assistants capable of responding to questions about the patient. To illustrate this, Comaniciu shared a video of a conversation with an AI radiologist assistant that responded to successive questions about the patient’s testing and findings.

Comaniciu presented several examples of AI-powered diagnostics for oncology, such as cancer risk prediction, brain tumor analysis and metastasis digitation and tracking, lung cancer screening via chest CT, assessment of pulmonary lesions via chest X-ray, breast cancer screening, prostate MRI, and analysis of cardiotoxicity. He also highlighted AI tumor fingerprinting, an approach under development to create a digital biopsy of molecular changes in tumors by integrating digital pathology, proteomic, metabolomics, and genomic profiling. Another fingerprinting approach is using an image-based deep neural network to optimize radiation therapy by stratifying patients as responders or nonresponders to guide individualized dosing (Lou et al., 2019; Randall et al., 2023).

Looking to the future, Comaniciu anticipated patients would have a “health digital twin,” which he characterized as a “lifelong, personalized, physiological model that is updated with each scan exam.” Simulations with a digital twin could inform treatment decisions, for example, or guide individualized preventive actions, he noted.

Building in Feedback Loops to Improve Quality of Care

Torbjörn Kronander, president and chief executive officer of Sectra AB, compared diagnostics to telecommunications in that both must detect signals in noise. Diagnostic testing detects signals of disease.

Kronander outlined some of the key features of IT systems for health environments. All relevant data should be available in the same place at the same time because clinicians do not have time to go back and forth among systems to find patient information, he said. Deep integration with AI is needed, and “AI should be applied both to radiology and pathology at the same time,” he said. The challenge, however, is that these data often reside in different systems. Uniform user interfaces can increase patient throughput, he noted, because moving to a user interface with a different format requires time to “reconfigure your brain” and adds “mouse miles” (computer mouse movements needed to engage with interfaces). Kronander added that unified tumor boards would help ensure that patient treatment is not delayed while discordant conclusions from pathologists and radiologists are resolved. Early identification and resolution of discordance among specialties can help the tumor board to work more efficiently, he noted.

Kronander explained that Sectra is working to integrate “radiology, pathology, and AI in one single IT system.” There is close integration with

information in the patient’s EHR, and Sectra is developing prototype structured reports that combine radiology, pathology, and concordance activities in one report sent to the EHR (he noted that the Sectra system currently only interfaces with the Epic EHR system). He added that Sectra is also collaborating with the University of Pennsylvania School of Medicine to develop a single user interface that incorporates pathology, radiology, and genomics.

Sectra’s approach to integrated diagnostics includes radiology, pathology, and omics data combined with current population health and probability data, Kronander said. He noted that probability data are often overlooked but are important for establishing a diagnosis. He explained that a key element of the Sectra integrated diagnostics model is feedback loops to support a learning system. Data from the structured diagnostic report in the EHR (where the final interpretation/outcome is entered) are also fed back to radiology, pathology, and omics functions and used to update population health and probability data.

The vision for integrated diagnostics, Kronander said, is “faster and better diagnosis, improved patient care, built in feedback loops [for] improving quality of health care, and at lower cost and less complexity.” He referred to Gambhir’s prediction that “[t]oday’s radiologists and pathologists will be replaced by diagnosticians plus AI” (Gambhir, 2018). Achieving this vision, Kronander said, will require “common basic training in decision theory for radiologists and pathologists.”

Kronander said that changes in reimbursement are also needed to achieve the vision of integrated diagnostics. He suggested that payers should reimburse clinicians for making the diagnosis rather than conducting specific diagnostic procedures. He acknowledged that such changes take time and observed that the U.S. system of reimbursement can be a barrier to the timely advancement of health care practice. Elenitoba-Johnson added that one challenge is coming to an agreement regarding “who is going to do the work and who is going to pay for it.” He emphasized that incentives need to be aligned to promote the adoption of integrated diagnostics into clinical practice.

Harnessing the Power of Data for Diagnosticians, Clinicians, and Patients

Nick Trentadue, director of laboratory and diagnostics informatics at Epic, identified some settings in which patient health care data are generated (see Figure 2). Traditionally, test results would be reported to the patient’s clinician, who would share them with the patient and discuss treatment options. Today, the results are also provided immediately to patients. Trentadue shared a screenshot of an actual report received by a patient awaiting cancer screening results. These reports, written by and for medical professionals, can be confusing and frightening to patients, he cautioned. Trentadue highlighted the need for tools that can better convey test results to patients.

___________________

²¹ An ontology, in the context of computer science, describes the relevant concepts, relationships, and specifications that are important for modeling a particular domain (Gruber, 2016).

___________________

²² See https://en.wikipedia.org/wiki/Explainable_artificial_intelligence (accessed January 25, 2024).

___________________

²³ See https://www.research.va.gov/programs/pop/lpop.cfm (accessed September 1, 2023).

²⁴ See https://mobile.va.gov/appstore (accessed May 26, 2023).

approximately 5,000 faculty physicians and 12,000 nurses and trains half the medical students and residents in California. It receives $2 billion in funding from NIH and brings in more than $16.5 billion in clinical operating revenue.

Butte said one original goal was to transform UCH into a single accountable care organization, facilitated by UC’s approach to data warehousing. The UCH-wide central data warehouse facilitates broad access to deidentified, structured clinical data from across all six academic health centers. A warehouse is still maintained at each of the six academic medical centers, which facilitates access to “deidentified, limited, and identified structured and unstructured clinical data” and additional information (e.g., images, genomes, notes). The central database has longitudinal, structured data going back to 2012. This includes data from 437 million patient encounters, 1.17 billion procedures, 1.5 billion medication orders, 48 million device uses, and 5.2 billion laboratory tests and vital signs, from nearly 9.2 million people. Data are merged with California state data and the California death index, Butte noted.

More than 100,000 patients with cancer receive care at a UCH comprehensive cancer center each year, noted Butte. The same UC-wide data warehouse contains demographic information, including information on a patient’s cancer diagnosis, insurance coverage and social risk, and more than 32,000 cancer genomic reports. Butte said these data can be queried to address a wide range of questions, such as whether patients who have a particular type of cancer with a specific gene mutation are receiving appropriate and equitable cancer treatment.

Butte explained that the UCH academic medical centers use the Observational Medical Outcomes Partnership Common Data Model. He noted that a systemwide committee meets every 2–4 weeks to harmonize data elements. Certain analyses require curation of unstructured clinical notes and data in the warehouses. Not having thousands of curators available, Butte described how UCH is looking at using an AI chatbot to curate deidentified clinical notes. He described one situation in which ChatGPT²⁵ was asked to summarize all cancer biomarkers in a complicated deidentified clinical note. The AI system was able to find and summarize data for all of the biomarkers, including one that the UCH oncology curators had missed.

UC researchers (including graduate students) who want access to UCH-wide data first write and optimize their queries in their campus system and then use a cloud-based tool to run the queries in the central UCH-wide warehouse. The goal is to facilitate “safe, respectful, regulated, responsible use of clinical data,” explained Butte. He said that data in the UCH-wide central data ware-

___________________

²⁵ ChatGPT stands for Chat Generative Pre-trained Transformer and was created by OpenAI. It is a large language model–based chatbot. See https://openai.com/blog/chatgpt (accessed September 1, 2023).

house are deidentified and can therefore be shared in compliance with the Privacy Rule²⁶ promulgated under the Health Insurance Portability and Accountability Act (HIPAA), and no institutional review board (IRB) approval is needed to conduct studies with these data. UCH has also determined that tumor mutation data are nonidentifiable and can be shared. He also noted that every patient is required to sign a terms and conditions of service document, which outlines how the university may, in compliance with HIPAA, use their data. A governance process is also in place to review external requests for the use of UCH data.

Training AI requires vast amounts of data, but competition among health systems is a barrier to data sharing across the country, Butte noted. While acknowledging the challenges, Butte contended that the goal is achievable, offering the NCATS/NIH National COVID Cohort Collaborative²⁷ as an example. Butte described how NCATS successfully motivated more than 100 health institutions to contribute deidentified data for all of their patients with COVID-19 to a central, third-party repository to support research on the disease. Data are publicly accessible through dashboards. Butte emphasized that this collaborative has led to nearly 500 data projects and 53 publications thus far. Butte pointed out that NCI does not require similar data sharing among their funded comprehensive cancer centers, and he advocated for the creation of a common data warehouse that could enable the development of precision medicine tools for patients with cancer.

Butte said that interoperability of health data is achievable, and a business need might drive the implementation of data sharing, but a culture that supports data sharing is also needed. Converting unstructured to structured data remains a challenge, but there is a potential role for AI. Other challenges noted by Butte include ongoing resistance to cross-campus clinical trials despite incentives and central IRBs and a lack of precision cancer care options for patients.

IMPROVING EVIDENCE GENERATION FOR INTEGRATED DIAGNOSTICS

Many speakers discussed approaches to generating evidence to inform and support evaluating, implementing, and scaling integrated diagnostics.

___________________

²⁶ The Privacy Rule, promulgated under the Health Insurance Portability and Accountability Act of 1996 (HIPAA), establishes national standards to protect individuals’ medical records and other individually identifiable health information (collectively defined as “protected health information”) and applies to health plans, health care clearinghouses, and those health care providers that conduct certain health care transactions electronically (see https://www.hhs.gov/hipaa/for-professionals/privacy/index.html).

²⁷ The National COVID Cohort Collaborative is also known as N3C. See https://ncats.nih.gov/n3c (accessed September 1, 2023).

adding that the Rush experience demonstrated “the feasibility of this approach for evidence dissemination and evidence generation for cancer screening.”

Building the Evidence Generation Infrastructure to Scale Precision Diagnostics

One challenge of precision diagnostics is advancing a technology born of “great science” to a product that can actually be used at scale, said Bradford Hirsch, head of Product and Implementation at Verily/Alphabet. He described two examples of taking precision diagnostics to scale from outside the field of oncology.

The first example was real-time screening for diabetic retinopathy using a deep learning system (Ruamviboonsuk et al., 2022). To bring it to scale, Verily first had to build a camera that could be used by clinicians globally to reliably capture images of the retina. Hirsch said that the camera was built by a multidisciplinary team that understood the relevant regulatory and reimbursement pathways. The camera can autonomously identify the retina in approximately 98.5 percent of patients, capture an image, and transmit it to the cloud for AI image assessment.

In a second example, he said that Verily designed a “study watch” for the collection of digital biomarker data from patients with Parkinson’s disease (Burq et al., 2022). Hirsch explained that despite many consumer wearables already available, it is generally not possible to access primary data from other devices—sensors and algorithms are frequently changed, for example—or customize the user experience. The watch developed by Verily takes remote measurements of motor function of individuals with Parkinson’s by alerting the wearer to complete a timed motor task and report the outcome through a survey on the device.

Hirsch explained that after product approval, there is a “data void,” so phase 4 postmarketing studies may be needed to gather additional data. This data void presents a challenge for precision diagnostics developers looking to take products to scale in clinical care. Verily is working to develop an evidence generation infrastructure with better continuous data collection after approval, closing the data gap between data collected during clinical research and data collected as part of clinical care.

Hirsch explained that Verily has restructured, bringing its research and care teams together under the same leadership team. Merging product development and implementation datasets can facilitate the creation of “products that can actually be used in practice,” Hirsch said.

Hirsch outlined five key elements of Verily’s approach to improve evidence generation:

___________________

²⁸ The Eastern Cooperative Oncology Group performance status scale helps define a patient’s overall ability to function after treatment. The Karnofsky Performance Status scale seeks to determine a patient’s impairment after treatment. See https://ecog-acrin.org/resources/ecog-performance-status/ and http://www.npcrc.org/files/news/karnofsky_performance_scale.pdf (accessed September 1, 2023).

the NCI Director’s Clinical Trials and Translational Research Advisory Committee. In using diagnostics, clinicians need to understand test performance characteristics, including the test’s sensitivity, specificity, positive predictive value, and negative predictive value;²⁹ the interpretability and actionability of the results; and the cost and convenience for the patient.

Meropol reviewed a few of the challenges to evidence generation for integrated diagnostics. Generally, the level of investment in evidence generation for diagnostics is lower than for drugs. In addition, technological advances are driving the development of numerous, increasingly complex diagnostic tools. As a result, clinicians are often faced with numerous testing options, many with inadequate clinical guidance, leaving them uncertain about what test to order or how to interpret the results.

He said that EHRs provide opportunities to gather evidence for integrated diagnostics, although they were not designed for research. EHRs are “a rich source of patient-level data” from multiple data sources, said Meropol. Much of the information in EHRs is digitized and available in real time. EHRs are embedded in the point-of-care workflow, which affords opportunities to provide clinical decision support in association with EHR data. Furthermore, data from EHRs tend to be more representative of the general population than data from clinical trials, which are from enrolled cohorts that trend toward “younger, healthier, Whiter, richer patients than the typical patient with cancer nationwide or worldwide,” he said. Meropol added that the U.S. Food and Drug Administration (FDA) has issued guidance addressing key considerations for using real-world data for clinical research and regulatory decision making (e.g., data quality, analytic approaches, fitness for purpose).³⁰

EHRs can be a platform for integrating evidence both from within and outside the EHR system, observed Meropol. This includes diagnostic testing reports, pathology and radiology images, genomic data, patient-reported data and data from wearables, insurance claims data, information on social determinants of health, and mortality data. He explained that mortality data are often missing or inaccurate in EHRs, but it is now possible to link to population registries to provide more accurate mortality data.

Meropol said the key capabilities for the use of EHRs to generate evidence include a “privacy framework that respects patients”; linkages with outside data sources, which might require tokenization of data; curation of both structured

___________________

²⁹ Sensitivity is the ability to identify a true positive, and positive predictive value is probability that a patient does have the disease if tested positive. Specificity is the ability to identify a true negative, and the negative predictive value is the probability that a patient does not have the disease if tested negative (Parikh et al., 2008).

³⁰ See https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence (accessed May 26, 2023).

and unstructured data; “careful application of ML and natural language processing to avoid misleading, inaccurate, or biased conclusions;” real-time data; and “context-specific characterization of data quality and fitness for purpose.”

Meropol emphasized the need for use cases and pointed out that not all data in the EHR are fit for purpose to address evidence gaps for integrated diagnostics, and information routinely collected in the course of clinical practice may be insufficient to address research questions. Passively collected data in EHRs may be characterized by missingness, confounding biases, selection biases, exposure variability, and variability in documented outcomes. One approach is to augment routinely-collected data with prespecified intentionally-collected data in the EHR to support prospective observational research studies (e.g., non-routine tests, testing at specific times or intervals, patient-reported outcomes).

Meropol described a case study in which Flatiron Health and collaborators used centralized, intentional EHR-based data collection and processing to support a prospective, observational study among patients with non-small-cell lung cancer.³¹ Circulating tumor DNA was obtained at prespecified time points to generate a dataset of integrated evidence that included genomic data, clinical data, and digital pathology, he said. One analysis, for example, compared the sensitivity of tissue-based genomic profiling to blood-based genomic profiling (Schwartzberg et al., 2022).

Meropol described the operational model implemented by Flatiron Health for gathering data from the EHR for prospective evidence generation. Data for research in the EHR may consist of routinely collected structured data (e.g., vital signs, laboratory tests, drugs), intentionally-collected structured data (not part of routine care but part of the routine workflow for research), and unstructured data (e.g., pathology reports, clinician notes). These data are transmitted from the EHR to a data warehouse through automated processes, with unstructured data first being processed by both humans and trained AI models. Data may then be linked to external sources of data, and data analytics are applied. These elements can come together in a platform approach to real-world evidence generation that Meropol said provides sites with the technology to integrate routinely and intentionally collected data to answer research questions with significantly less site burden than traditional prospective clinical studies (Bourla and Meropol, 2021) (see Figure 4).

Meropol said that many tools to support evidence generation for integrated diagnostics exist and increasing adoption of these tools is occurring. ML tools are available to support automated patient ascertainment. EHR systems have the capability to embed data collection forms for research in routine clinical workflows. And methods exist for the processing of struc-

___________________

³¹ See https://flatiron.com/resources/case-study-prospective-clinicogenomic and https://clinicaltrials.gov/ct2/show/NCT04180176 (both accessed May 26, 2023).

___________________

³² See https://www.aacr.org/professionals/research/aacr-project-genie/ (accessed September 5, 2023).

³³ See https://www.cbioportal.org/ (accessed February 2, 2024).

³⁴ See https://www.aacr.org/professionals/research/aacr-project-genie/news-updates/ (accessed October 26, 2023).

³⁵ PRISSMM is a cancer data modeling system; it stands for pathology, radiology, imaging, signs and symptoms, tumor markers; and medical oncologist assessments. See https://www.mskcc.org/research-advantage/support/digital-health-projects/prissmm-cancer-data-modeling-system (accessed September 5, 2023).

such as prior treatment history, current therapy, progression-free survival (PFS), and overall survival (Lavery et al., 2022). This work is being funded by the 10 biopharmaceutical partners, and genomic and clinical data for 16,000 patients have been curated so far. Sawyers showed data on checkpoint inhibitor treatment for patients with lung cancer, which demonstrated that determining PFS using PRISSMM is comparable to doing so with the RECIST criteria, which assess disease status based on imaging data.

Sawyers shared several examples of how Project GENIE data are being used, such as to generate external control cohorts for rare disease research (Scharpf et al., 2022; Smyth et al., 2020). In one case, GENIE data were combined with other datasets to create a control cohort that was submitted along with the data from single arm studies of sotarasib³⁶ to support FDA regulatory review and approval of the treatment for a rare type of lung cancer (Scharpf et al., 2022).

Project GENIE data are also being used to explore race and ethnicity differences across cancer genotypes, Sawyers said. One example he mentioned was a study of the frequencies of BRCA2³⁷ mutations among Black versus White men with prostate cancer. In another example, Project GENIE data are being used to better understand population-specific miscalling of somatic³⁸ variants, which can inflate somatic mutation estimates among those with non-European ancestry. Studies are also underway to determine genetic ancestry using sequence panels in the registry, compare that with self-reported race, and conduct admixture analysis. Robert Winn, director of the Virginia Commonwealth University Massey Comprehensive Cancer Center, asked about the study of genetic ancestry. For example, triple negative breast cancer is known to disproportionately affect Black women, and Winn referenced a genetic ancestry study that found differences based on Eastern African versus Western African ancestry. Sawyers responded that the intent is to delve into these types of differences with the sequencing panels available. He said that in the absence of whole-genome sequencing of germline DNA, sequencing panels of around 500 genes can provide a remarkable amount of accurate information. Levy

___________________

³⁶ Sotarasib is a KRAS inhibitor approved by FDA as a lung cancer treatment in 2021. The KRAS gene regulates proliferation signals, and mutations can cause unregulated growth, leading to cancer. See https://www.cancer.gov/news-events/cancer-currents-blog/2021/fda-sotorasib-lung-cancer-kras (accessed September 5, 2023).

³⁷ BRCA2, or BReast CAncer gene 2, mutations are associated with increased cancer risk. See https://www.cancer.gov/about-cancer/causes-prevention/genetics/brca-fact-sheet (accessed September 5, 2023).

³⁸ Somatic refers to cells that are not part of the germ line and therefore not passed to future generations. See https://www.genome.gov/genetics-glossary/Somatic-Cells (accessed September 5, 2023).

added that Foundation Medicine has also developed an algorithm based on geographic ancestries. Lennerz noted the lack of consensus on genetic markers based on geographic ancestry data that could be used for validation. Levy noted that even though there are data gaps, genetic geographic ancestry can be useful for increasing diversity among clinical trial populations. Sawyers acknowledged the challenges and added that the human genetics community has achieved “a broader representation of different parts of the world in terms of genome databases” than the cancer community.

A challenge going forward, Sawyers said, is ensuring that data in clinicogenomic registries are generalizable. Project GENIE currently includes four institutions in Europe, and one in Asia will be joining soon. It is also working to increase representation of people of non-European backgrounds in the registry by adding new contributing sites and collaborating with organizations that have genomic data from diverse populations. Sawyers noted that there are barriers to inclusion of persons of non-European ancestry in clinic-genomic registries that need to be overcome, such as meeting the infrastructure requirements for centers to contribute to registries (e.g., ability to deliver the type and quantity of data needed, full informed consent processes, and appropriate data stewardship) and building the trust needed to increase participation of diverse communities in research. Levy added that there are specific challenges for including data from non-U.S. institutions, including different approaches to regulation and ensuring patient privacy. The diversity of the data in Project GENIE is based on the diversity of the patient populations of the contributing institutions, and institutions that serve more diverse populations are also those that often do not yet have the infrastructure needed to participate. Sawyers said that Project GENIE generally does not have the resources to support this infrastructure but can sometimes assist with start-up costs. He also noted that some centers that do have the infrastructure but prefer not to participate.

Mark Stewart, vice president of science policy at Friends of Cancer Research, raised the issue of variability in the performance of different diagnostic assays with the same intended use and how that might affect harmonization and interpretability of data from different institutions. Levy responded that Project GENIE datasets include data from different assays, and the ability to ask a particular question will be limited by whether a gene was included in particular assays. Another issue, discussed by Trentadue, is the need for a clear ontology for diagnostic procedures. Levy said, for example, that testing menus from different institutions might call the same diagnostic test “a right-sided mammogram, a right-side diagnostic mammogram, right-sided mammo, or R mammo.”

___________________

³⁹ See https://ncorp.cancer.gov/ (accessed September 5, 2023).

⁴⁰ See https://www.fqhc.org/what-is-an-fqhc/ (accessed January 27, 2024).

Informational Science and Engineering at the University of Florida. AI models in diagnostics are increasingly computationally complex, with an emphasis on prediction speed and quality but with less transparency. These models can lead to unintended consequences, including shortcut learning (unintended learning from the training materials), which may contribute to unexpected failure of algorithms in broader clinical practice.

Thai said another concern is the potential for bias. Training datasets are collected and annotated by humans, making them subject to human biases. Encoded biases in the dataset are amplified during training, leading to biased outputs. “In some cases, the output data becomes the training data for the next model,” she said, leading to an ongoing cycle of bias amplification.

Thai noted the patient privacy and data security concerns associated with ML and described several types of adversarial attacks that can occur in AI design. The training phase can have “poisoning” of the training data with data intended to alter model performance. In the inference phase, when the predictive model is deployed, adversarial attacks can include “membership inference,” an attempt to reverse engineer the model to predict what data were included in the training set; “model extraction,” in which the outcome is used to steal the functionality of the model; and “adversarial examples,” which are noisy data added to the input with the intent to cause the model to misclassify outcomes.

To address these concerns, Thai listed six key elements to design ethical AI models: they should be transparent and understandable, inclusive, responsible, impartial and unbiased, trustworthy and reliable, and able to ensure data security and user privacy. Thai also emphasized the need to ensure that training datasets are appropriately balanced with respect to population diversity and assess whether the model, as designed, applies equally to everyone—a key aspect of the “inclusion” element of ethical AI.

Thai also said that designing ethical AI is not just the domain of computer science but requires multidisciplinary collaboration. This approach focuses on data security and patient privacy, explainable AI, and fairness in the development of an AI model with high predictive accuracy (Phan et al., 2019, 2020). She noted that there is little value to developing a secure, fair, explainable model that is inaccurate and explained that these elements are overlapping. For example, data security and patient privacy are important to explainable AI (e.g., blocking new avenues of adversarial attacks) and fairness (e.g., minimizing risk of exposure of sensitive information, such as age, sex, and race). It is also important to be alert to the creation of new problems and address them in the development process. For example, adversaries can query the model to receive an explanation of how the AI derived the prediction and then exploit that information to devise an attack. The challenge, Thai said, is how to provide useful explanations to users without enabling adversarial attacks (Nguyen et al., 2022).

to those improved outcomes. He added that the work of the DMTs has demonstrated significant value for the hospital system.

Having implemented a DMT approach at UTMB and other institutions, Laposata is working on efforts to create a national DMT, in which a patient’s clinician would reach out to a local diagnostic physician with questions, who would provide answers based on their knowledge and consult with national-level DMTs for further assistance as needed. The local physician would provide the information from the national DMT to the patient’s clinician, who can then act on the experts’ recommendations. If done well, Laposata said, a national DMT model could facilitate rapid and accurate diagnoses that improve patient outcomes.

Multimodal Data Integration for Research and Clinical Practice

Sohrab Shah of MSK explained that computational research can facilitate insights from real-world data, multimodal data integration, cohort derivation and comparison, and patient stratification and predictive models (Boehm et al., 2022a, 2022b). MSK has long invested in developing platforms for cancer data science, Shah said, including the following:

• cBioPortal for cancer genomics datasets⁴¹
• MSK MIND (Multimodal Integration of Data)⁴²
• OncoKB (a precision oncology knowledge base)⁴³
• Warren Alpert Center for Digital and Computational Pathology⁴⁴

Shah said that the understanding of cancer biology is informed by experimental data on cancer progression and drug resistance, the tumor micro-environment, cellular phenotypes, interactions among tumor cells and immune cells, and multiomics data. He said that applying computational oncology approaches to these data can facilitate the development of new cancer treatments and integrated diagnostics.

As proof of principle for clinical application, Shah described an example of multimodal real-world data integration to predict response to immunotherapy in patients with non-small-cell lung cancer. The lung tumor experts at MSK were facing challenges predicting who would respond to an immune

___________________

⁴¹ See https://docs.cbioportal.org/about-us/ (accessed September 5, 2023).

⁴² See https://www.mskcc.org/research-programs/msk-mind-multi-modal-integration-data (accessed September 5, 2023).

⁴³ See https://www.oncokb.org/ (accessed January 25, 2024).

⁴⁴ See https://www.mskcc.org/departments/pathology-laboratory-medicine/warren-alpert-center-digital-and-computational-pathology (accessed September, 5, 2023).

checkpoint blockade therapy. Shah and colleagues developed a computational workflow and ML approach to extract and integrate genomic, histology, and radiology data to provide a prediction of a patient’s response to therapy (Vanguri et al., 2022). The development process was labor intensive, said Shah; a thoracic pathologist reviewed and labeled all tumor regions in digitized immunohistochemistry images. “We then started to compute the topological features of the staining properties within each of the tumor regions that she labeled,” he explained. This took approximately 1.5 years, with a comparable process for the radiology images. Shah said that the integrated multimodal data approach provided superior predictive performance compared to several predictive approaches that involved only one modality (Boehm et al., 2022a; Vanguri et al., 2022; Vázquez-García et al., 2022).

Shah also described a study integrating multiomic data to elucidate the natural history of disease for ovarian cancers, particularly the relationship between genomic DNA damage and the immune response (Vázquez-García et al., 2022). This was also a labor-intensive, multidisciplinary process involving surgical collection of tumor tissue, whole genome sequencing, single-cell RNA sequencing, digital pathology, and phenotypic profiling of immune cells.

“Multimodal data integration is a very powerful approach for both real-world and experimental data interpretation,” Shah summarized, and he shared several lessons. It requires both clinical expertise and computational rigor, and he emphasized the need for multidisciplinary collaboration. He suggested that large-scale retrospective multimodal studies are needed to inform integrated diagnostics but also noted that obtaining data for these can be challenging. He explained that multimodal data integration at scale requires data engineering, because diverse types of data need to be drawn from disparate sources into a centralized repository that can be queried and used for ML models. Reproducibility in ML is essential, but Shah noted that can be difficult and different institutions need to validate the models. Finally, Shah noted that the field faces a talent “bottleneck,” and more data scientists need to be recruited to the field of cancer research and care.

Implementing Integrated Diagnostics into Precision Oncology Care

David Dorr, chief research information officer and vice chair of Medical Informatics and Clinical Epidemiology at Oregon Health & Science University (OHSU) School of Medicine, defined health informatics as the science of the use of data, information, and knowledge to improve health.⁴⁵ Dorr discussed ways in which the multidisciplinary field of health informatics

___________________

⁴⁵ See https://amia.org/about-amia/why-informatics/informatics-research-and-practice (accessed January 25, 2024).

seeks to understand how users interact with data and to mitigate cognitive and data biases in the use of these data. From a cognitive perspective, several key elements determine whether and how people act on computer-aided decision support recommendations for patient care, and Dorr referenced the affect-integration-motivation and attention-context-translation framework (Nahum-Shani et al., 2022).⁴⁶ Key issues that affect clinician uptake of findings from digital interventions such as integrated diagnostics include trust, understanding, timing, options, prioritization, adjudication and annotation, and actionability. Given the many things that compete for a clinician’s time and attention, Dorr emphasized that a synthesis of findings from an integrated diagnostics tool needs to make clear what the most important information is, why it is important for their patient at that time, and what actions can be taken.

Dorr described a study comparing algorithmic risk scoring to clinical intuition in predicting which patients were most likely to be hospitalized within the coming year (Dorr et al., 2021). The study found that algorithms were more accurate than clinical intuition for risk prediction and that clinician adjudication of algorithmic risk scores improved performance further, which he said was associated with confidence and trust in the risk scoring process.

Dorr pointed out that most algorithms rely on EHRs and other real-world data sources, which may contain errors or conflicting information. For example, diagnoses appear in multiple EHR locations and can be inaccurate or missing, impacting predictions (Martin et al., 2017). Dorr added that missing data are not necessarily random and can be associated with inequities.

Data may also contain biases that reflect societal biases, Dorr continued, and algorithms can perpetuate biases depending on how they are designed. Dorr cited an example in which an algorithm predicted that Black and White patients have the same level of risk for future health care use (Obermeyer et al., 2019). However, Black patients experienced more health problems that would likely necessitate higher future health care utilization. In this case, the underlying bias was inequitable access to health care: the algorithm predicted future risk based on spending data, which was more reflective of a patient’s access to care and not on their health status.

Integrating advanced algorithms into the clinical workflow also presents challenges. Dorr discussed the role of implementation science in promoting the uptake of integrated diagnostics and preventing unintended consequences. He mentioned the Consolidated Framework for Implementation Research as an example of one effective approach (Damschroder et al., 2022).

___________________

⁴⁶ The affect-integration-motivation and attention-context-translation framework is intended to “provide recommendations for designing strategies to promote engagement in digital interventions and highlight directions for future research” (Nahum-Shani et al., 2022).

ment decisions). He offered three strategies to enhance trust in the technology and improve acceptance and adoption (Choudhury, 2022):

• Establish shared or distributed accountability based on risk and intended use.
• Take human factors into consideration (e.g., trust, perceptions, workload).
• Implement need-based integration to help end users to understand how AI is a tool that can help them improve effectiveness and efficiency.

Risk Communication and Decision Support

Patient perception and understanding of health risks vary, said Mary Politi, professor in the Department of Surgery at Washington University School of Medicine. Clinician abilities to explain clinical uncertainty to their patients and manage patient care in the context of uncertainty can also vary. She referenced an interview-based study that identified four categories of strategies clinicians use for uncertainty management, focused on ignorance, uncertainty, response, and relationships (Han et al., 2021). The first two approaches seek to reduce the uncertainty, and the last two seek to cope with its effects. Politi centered her remarks on the relationship-focused tactic of sharing information about clinical uncertainty with patients. She quoted a clinician in the study who discussed the science versus the art of medicine and the need to embrace the uncertainty and share information with the patient to help them make personalized choices about their care.

Politi described the three-talk model of shared decision making: a “team talk” to establish the clinician–patient partnership in treatment decisions; an “option talk” using risk communication strategies to inform patients about their care options; and a “decision talk” to incorporate patient needs and preferences in the care decision (Elwyn et al., 2017). The option talk includes risk prediction models incorporating the results of diagnostic testing. Politi summarized some central tactics of communicating risks, benefits, and uncertainty to patients during the option talk:

• Using numbers. Avoid imprecise terms, such as “low,” “moderate,” or “high,” when discussing risk. “Without numbers, people will underestimate their risk and overestimate their benefit,” Politi said.
• Using frequencies and keeping the denominator consistent. Make it easy for patients to make comparisons (e.g., do not present the likelihood of one outcome as 1 in 10 and another as 1 in 12).
• Using visuals. Pictures can help to convey risk information more clearly. Many risk communication studies have suggested the use of icon arrays.

___________________

⁴⁷ This tool is publicly available from the Melanoma Institute Australia at https://www.melanomarisk.org.au/SNLForm (accessed May 26, 2023).

⁴⁸ See https://breastchoice.wustl.edu/ (accessed September 5, 2023).

⁴⁹ The A1C test is used to measure a patient’s average blood sugar levels and can be used to diagnose or monitor prediabetes and diabetes. See https://www.cdc.gov/diabetes/managing/managing-blood-sugar/a1c.html (accessed September 5, 2023).

___________________

⁵⁰ See https://www.ohdsi.org/ (accessed September 5, 2023).

810 million patients. Dorr explained that variation in algorithm performance is to be expected and provides opportunities for learning. Shah agreed that federated learning could be an option for algorithm validation and reproducibility studies, in which the algorithm is trained on datasets at multiple institutions and then the models are integrated. Shah cited a recent report of federated learning for predicting response to a treatment for breast cancer (Ogier du Terrail et al., 2023).

Regina Barzilay, MacArthur Fellow and School of Engineering Distinguished Professor for AI and Health in the Department of Electrical Engineering and Computer Science at Massachusetts Institute of Technology’s Computer Science & Artificial Intelligence Laboratory, suggested that another reason for lack of reproducibility is that algorithms in health care are not as robust as those in other contexts. Existing software is often adapted for use with medical data without “the same degree of mathematical sophistication and engineering.” As an example of a robust algorithm, she noted that “it is really hard to find a place where your face will not be recognized” by an iPhone’s facial recognition software.

Barzilay also noted the role for publishing standards that foster reproducibility and that some journals expect demonstration of reproducibility across institutions as a requirement for publication.

Overcoming Design and Use Challenges

Many speakers suggested potential opportunities for overcoming the challenges related to designing and using integrated diagnostics in precision cancer care. Robison, referencing the challenge of EHR organization of information by type (e.g., laboratory data are in one location, images are in another) suggested that the data visualization platforms should be contextually aware, presenting the data that are needed in that moment for the decision. Shah agreed that organization should be patient centric rather than by data type. He recalled Comaniciu’s discussion of digital twinning and said MSK is exploring the concept of a user interface built around a digital patient, but implementation challenges remain, such as how data are weighted and selected for context-relevant inclusion in the presentation. Dorr said that some type of “annotation or adjudication of what is most important” is needed to determine “what data should be prioritized at what time.”

Dorr highlighted the need for research on understandability and actionability of information. How do clinicians interact with the volumes of data related to their patients, especially those with complex conditions, and what is it that they actually look at? He said health care algorithms need to be adapted to include data elements that clinicians use. “There is real harm that’s done when data are missed,” he cautioned. He suggested looking to the field of

industrial engineering to understand how to insert information into different workflows.

Politi observed that graphics on decision aids can be very complicated and include animations and other features. While visually appealing, she explained that some more complex presentations can be distracting for the patient receiving the information. She advocated for clear, simple diagrams that can be personalized as needed.

Choudhury suggested developing AI algorithms to assess whether a patient is at risk for severe depression or suicidal ideation when a clinician is delivering a diagnosis. This could inform how care is delivered, including provision of additional mental health support.

REGULATORY OVERSIGHT AND INSURANCE COVERAGE OF INTEGRATED DIAGNOSTICS

Several speakers discussed the importance of achieving regulatory clearance or approval for clinical use and establishing coverage and reimbursement mechanisms for the adoption of integrated diagnostics.

Pathways for Regulatory Review of Integrated Diagnostics

Reena Philip, associate director for Biomarkers and Precision Oncology at the FDA Oncology Center of Excellence, provided an overview of the current FDA review framework for IVDs, radiological devices, and AI-based digital pathology and radiological devices.

Philip explained that the regulatory pathway for all medical devices begins with the submission of an application for premarket review of safety and effectiveness by FDA’s Center for Devices and Radiological Health (CDRH). Applications are processed differently according to the device’s risk classification (Box 3). Upon clearance or approval, the device may enter the market for clinical use. Safety and effectiveness monitoring continues through postmarket surveillance activities including required reporting of serious and adverse events associated with device use. The FDA Total Product Life Cycle database integrates premarket and postmarket data about medical devices, including about adverse events and recalls.⁵¹

___________________

⁵¹ See https://www.fda.gov/about-fda/cdrh-transparency/cdrh-transparency-total-product-life-cycle-tplc (accessed May 26, 2023).

___________________

⁵² A companion diagnostic, usually an IVD, is a medical device that provides necessary information on how to use a drug or biologic product safely and effectively. See https://www.fda.gov/medical-devices/in-vitro-diagnostics/companion-diagnostics (accessed January 28, 2024).

___________________

⁵³ See https://www.fda.gov/medical-devices/digital-health-center-excellence/digital-health-terms (accessed May 26, 2023).

⁵⁴ See https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices (accessed May 26, 2023).

⁵⁵ See https://paige.ai/ (accessed September 5, 2023).

locked algorithm, with its performance dependent on the training dataset. She said that product sponsors needed to ensure sufficient data to support the intended uses and that the intended patient population is represented in the validation dataset.

In response to a question, Philip said that, to her knowledge, all AI-based devices reviewed by FDA thus far have been locked algorithms. In reviewing adaptive algorithms, she said it would be necessary for FDA to understand up front how performance of one deemed safe and effective could be monitored after clearance or approval to ensure ongoing safety and effectiveness in the marketplace. She added that research is needed to determine how to perform this monitoring.

Opportunities and Challenges

AI-based medical devices can use real-world data to continuously learn and improve and have the potential to “transform the delivery of health care,” Philip said. But challenges remain, including creating the “large, high-quality, well-curated datasets” needed to train models; explaining AI algorithm decision making (i.e., “black box”⁵⁶ approaches); identifying and eliminating biases in models; and ensuring transparency for users.

Philip highlighted several FDA activities to address these challenges. The FDA–NIH Joint Leadership Council Working Group on Next Generation Sequencing and Radiomics explored the development of reference materials to support the validation of next-generation sequencing tests and the use of AI and ML to interpret next-generation sequencing and radiomics data. She referred participants to the highlights of a workshop in September 2021.⁵⁷ Related initiatives with AI-focused activities include the following:

• Medical Device Innovation Consortium Somatic Reference Sample Initiative,⁵⁸
• Digital Health Center of Excellence-AI and Machine Learning Software as a Medical Device Action Plan,⁵⁹

___________________

⁵⁶ See https://www.scientificamerican.com/article/why-we-need-to-see-inside-ais-black-box/ (accessed November 1, 2023).

⁵⁷ See https://dctd.cancer.gov/NewsEvents/20211122_NCI_Hosts_FDA_NIH_Workshop.htm (accessed May 26, 2023).

⁵⁸ See https://mdic.org/wp-content/uploads/2022/10/MDIC-Initiative-SRS_CS_FINAL-Web.pdf (accessed January 25, 2024).

⁵⁹ See https://www.fda.gov/news-events/press-announcements/fda-releases-artificial-intelligencemachine-learning-action-plan (accessed January 25, 2024).

___________________

⁶⁰ See https://www.fda.gov/media/171713/download (accessed January 25, 2024).

⁶¹ See https://datascience.cancer.gov/data-commons/repositories (accessed January 25, 2024).

⁶² See https://precision.fda.gov/ (accessed March 6, 2024).

⁶³ See https://precision.fda.gov (accessed May 26, 2023).

⁶⁴ See https://www.ama-assn.org/about/cpt-editorial-panel/cpt-code-process (accessed January 28, 2024).

funding to support this research. Malin responded that payers contribute to the funding for the Patient-Centered Outcomes Research Institute.⁶⁵

ENSURING PATIENT ACCESS AND PROMOTING HEALTH EQUITY

Many speakers discussed mechanisms to enable broad, equitable patient access to safe and effective integrated diagnostics, particularly in community-based settings of cancer care. The focus needs to be on “making sure that there is health equity in every one of our novel innovations, including integrated diagnostics” emphasized Beth Karlan, the Nancy Marks Endowed Chair in Women’s Health Research, and director of Cancer Population Genetics at the University of California, Los Angeles, Jonsson Comprehensive Cancer Center.

Developing Equitable AI Algorithms

Barzilay discussed challenges and solutions for developing safe and equitable AI algorithms. She pointed out that while there have long been disparities in health and biases in standard prediction models, there are several reasons to pay particular attention to the potential for bias in deep learning models. First, these models are “data hungry,” requiring a training set of 50,000–100,000 images, she said. For many clinical specialties, this volume of training data will not be available. Deep neural networks are also poor at handling distributional shifts in data. As an example, she showed how the performance of a model trained to recognize white numbers on black backgrounds declines when it is presented with colored numbers on different backgrounds. In addition, deep learning models can learn and perpetuate biases in the training dataset. She described how an image recognition model predicted that a monkey with a guitar was a human because the guitars it saw in the training data were always associated with humans.

“Most diagnostic tasks are beyond human prediction capacity,” Barzilay said. A human cannot look at an image and predict when someone will get cancer or have a recurrence, for example. She explained that this presents a problem for validating ML risk prediction models. She described three novel algorithmic solutions used to develop two validated risk prediction models, Mirai and Sybil, to ensure the models are robust. Mirai uses a patient’s screening mammogram to predict their risk of breast cancer within 5 years (Yala et al., 2021, 2022a, 2022b), and Sybil uses low-dose CT chest scans to predict a patient’s risk of developing lung cancer within 6 years (Mikhael et al., 2023).

___________________

⁶⁵ See https://www.pcori.org/ (accessed January 28, 2024).

___________________

⁶⁶ See Bao, Y., and R. Barzilay. 2022. Learning to Split for Automatic Bias Detection. https://arxiv.org/pdf/2204.13749.pdf (open access/pre-print repository document) (accessed May 26, 2023).

⁶⁷ See Fisch et al. 2022. Calibrated Selective Classification. https://arxiv.org/pdf/2208.12084.pdf (open access/pre-print repository document) (accessed May 26, 2023).

___________________

⁶⁸ See https://www.cancer.gov/research/infrastructure/cancer-centers/find (accessed May 26, 2023).

tices within organizations. He suggested that that a new program or institute might be needed to drive progress by organizing the many laboratories and people around “a shared mission, a shared language, shared resources, [and] shared time lines.”

Oyer referred to a previous National Cancer Policy Forum workshop in which participants discussed the growing challenges facing the cancer careforce and potential solutions focused on improving patient experience and outcomes, the capacity and effectiveness of caregivers, and the efficiency, effectiveness, and resilience of clinicians (NASEM, 2019; Takvorian et al., 2022). Oyer said these solutions include “leveraging technology to share and integrate information to improve outcomes, efficiency, and reach,” and this includes digital technology and AI.

Patients receive care from different teams of clinicians, at different locations, using different tools, as they traverse the care continuum from diagnosis to treatment to surveillance and recovery, Oyer noted. He stressed the importance of precise, accurate, and timely bidirectional communication about integrated diagnostics across this continuum.

Oyer offered several suggestions to activate integrated diagnostics in community settings, such as:

• Establishing networks of care that share competence and collaboration across locations.
• Building infrastructure for cancer imaging networks to share expertise.
• Providing reports to clinicians that are timely, understandable, and usable in the clinic and at the bedside.
• Seamlessly incorporating integrated diagnostic reports into the EHR, and automatically interface them with widely available and widely used cancer care guidelines.

Oyer added that in order to ensure equity and representation, members of the cancer research community need to collaborate and build knowledge by collecting and sharing patient data, including the creation of a data bank, similar to the collection of tissue for a tissue bank.

Oyer also offered several suggestions to facilitate equitable access to precision diagnosis and clinical trials that improve outcomes for every patient with cancer in every community, such as expanding the capacity and representativeness of the oncology workforce, including diagnosticians. He added that digital tools can be used to augment capacity and that training is needed to ensure that caregivers can integrate these tools. Oyer also called on NCI and Congress to establish a “hub-and-spoke” system for the delivery of cancer care and to mandate EHR interoperability using mCODE.

___________________

⁶⁹ See https://www.cancer.gov/about-cancer/treatment/drugs/docetaxel (accessed September 5, 2023).

tor testing made it possible to stop treatment with tamoxifen⁷⁰ for those who would not benefit from it. Brawley also emphasized the need for large clinical studies with subset analysis to elucidate the distribution of various markers among different types of populations (i.e., not only by race). Integrated diagnostics supports evidence-based care and prevention and “the rational practice of medicine,” he concluded.

Understanding and Addressing Disparities in Access and Use

Panelists were asked what could done to reduce disparities in access to diagnostics. Brawley and Oyer both said that the medical system cannot solve this problem alone. Many health disparities are rooted in “the fabric of society,” Brawley explained, and both agreed that political action is needed, but political will is lacking. Oyer said clinicians can work to bring new tools and technologies to their communities and “provide [care] to the patient in front of us, without bias, each and every day.” He advocated for a national universal health care system to help achieve this goal. Barzilay reiterated the need for diverse datasets that are representative of the whole population and stressed the importance of transparency when publishing studies based on biased historical datasets. She also said that the National Science Foundation (NSF) and NIH should dedicate sufficient resources to algorithm development to ensure that forthcoming AI systems perform well across different populations, especially minority populations.

Winn asked what infrastructure and workforce are needed to address the “data deserts” and how a “two-tiered AI system” might be avoided so that everyone receives the full benefit of the technology, not just those with access to an academic medical center. Brawley said that for many clinical settings, especially safety net hospitals, incorporating new technology and services can make it even harder to meet current needs of the community. He reiterated his example of how implementing lung cancer screening in safety net hospitals would benefit some people but would make the wait time for any type of CT scan longer for everyone, reducing quality of care. Brawley stressed that resources need to be provided for screening so that current services are not undermined.

Shulman pointed out that most smaller hospitals in rural areas are not part of a hub-and-spoke or other network, and these hospitals often have the largest disparities in care. Drawing on Winn’s comment about data deserts, Oyer and Shulman noted that said these rural areas are “deserts” outside of the catchment areas of NCI-designated cancer centers and called for a national effort to

___________________

⁷⁰ See https://www.cancer.gov/publications/dictionaries/cancer-terms/def/tamoxifen-citrate (accessed September 5, 2023).

address these gaps. Shulman suggested that large academic medical centers have a responsibility to partner with these hospitals to improve care. Oyer suggested a role for NCI-designated cancer centers in expanding access to programs and digital resources at these hospitals. He called for bold and transformative solutions to ensure that these parts of the country have access to care.

Barzilay said there is a role for AI in taking a “more algorithmic approach of assessing risk and acting upon it.” Oyer said that health-related social needs should also be considered in patient risk assessment. They discussed the role of risk assessment in managing utilization of integrated diagnostics, ensuring access for patients who need a diagnostic test while reducing unnecessary testing of those who do not, especially in the face of limited resources for testing. Malin noted the importance of taking the intended use of the diagnostic test results into consideration. She agreed that not everyone needs to have every test done and added that unnecessary testing is costly for both the health system and the individual patient, emphasizing that the costs for patients include time spent on testing. Malin added that patient preferences need to be taken into consideration when deciding how comprehensive testing needs to be. For example, certain tests might not be necessary if a patient does not want to pursue a particular therapy.

CURRENT FEDERAL INITIATIVES IN PRECISION CANCER CARE

Two keynote speakers discussed current initiatives at NCI and the Advanced Research Project Agency for Health (ARPA-H) to advance precision oncology care.

NCI Precision Medicine Cancer Trials

Lyndsay Harris, associate director of the Cancer Diagnosis Program in the NCI Division of Cancer Treatment and Diagnosis, described the NCI Molecular Analysis for Therapy Choice (NCI-MATCH) clinical trial (Box 4).⁷¹ NCI-MATCH is a precision medicine trial that examines patient treatment approaches based on the molecular profiles of their tumors, said Harris. Data from the NCI-MATCH trial are being used to identify and evaluate potential diagnostic biomarkers.

NCI is also interested in studying how socioeconomic and environmental factors correlated with health disparities are associated with tumor and clinical

___________________

⁷¹ See https://www.cancer.gov/about-cancer/treatment/clinical-trials/nci-supported/nci-match (accessed May 26, 2023).

___________________

⁷² See https://ecog-acrin.org/clinical-trials/eay191-combomatch/ (accessed February 2, 2024).

⁷³ See https://www.cancer.gov/about-cancer/treatment/nci-supported/combomatch (accessed February 2, 2024),

⁷⁴ See https://deainfo.nci.nih.gov/advisory/fac/1019/Doroshow.pdf (accessed February 2, 2024).

⁷⁵ See https://www.whitehouse.gov/briefing-room/speeches-remarks/2022/03/18/remarks-by-president-biden-before-a-discussion-with-researchers-and-patients-on-advanced-research-project-agency-for-health-arpa-h/ (accessed May 26, 2023).

⁷⁶ Monarez said that ARPA-H is actively seeking people to be program managers and fill other agency positions, and she encouraged workshop attendees to visit https://arpa-h.gov/careers/work-with-us/ for more information (accessed May 26, 2023).

___________________

⁷⁷ See https://sharing.nih.gov/data-management-and-sharing-policy/about-data-management-and-sharing-policies/data-management-and-sharing-policy-overview#after (accessed January 31, 2024).

⁷⁸ See https://datascience.cancer.gov/data-commons (accessed September 5, 2023).

of the White House Office of Science and Technology Policy.⁷⁹ The intent is that all data will be “available to those who would benefit from its utilization.” “Without data, you can’t innovate,” she concluded.

REFLECTING ON THE WORKSHOP DISCUSSIONS

In the final session, the moderators of each session reflected on key points highlighted by speakers, including opportunities to advance the use of integrated diagnostics in cancer care (see also Boxes 1 and 2).

Envisioning the Future for Integrated Diagnostics

Hricak emphasized that “diagnostics are moving toward data science, and there is a need for data integration.” She suggested that the scope of integrated diagnostics remain flexible and patient centered, noting that these go beyond a clinical decision support system or data display dashboard. Many speakers highlighted that funding for integrated diagnostics can be a challenge, and it can be difficult to demonstrate a return on investment in the short term, Hricak said. Elenitoba-Johnson called for the development of policies and incentives to promote the adoption of integrated diagnostics in clinical practice. Hricak said other considerations highlighted by many speakers included improving interoperability of data systems across clinical disciplines; implementing data standards, synoptic operative reports, and a lexicon to convey the degree of diagnostic certainty; facilitating collaboration to transition image annotation and segmentation from manual to automated; fostering a culture of data sharing; restructuring reimbursement to cover the diagnosis versus diagnostic testing procedures; scaling up precision diagnostics to reach broader populations; and leveraging institutional governance to adopt integrated diagnostics within clinical care.

Highlighting Insights from Academia, Industry, and Health Care Organizations

Nancy Davidson, executive vice president for Clinical Affairs at the Fred Hutchinson Cancer Center and Raisbeck Endowed Chair for Collaborative Cancer Research at the University of Washington, highlighted perspectives from representatives of academic medical centers, industry, and large health care organizations. Several speakers stressed the need to facilitate the integra-

___________________

⁷⁹ See https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf (accessed February 1, 2024).

tion of data from across disciplinary or organizational silos, she said, while others suggested that integrated diagnostics could better manage and communicate diagnostic testing results. To promote adoption, several speakers suggested conducting analyses to assess the value of integrated diagnostics in achieving high-quality, efficient care, which could provide evidence to support payment models and insurance coverage.

Davidson said that many speakers highlighted the importance of training the next generation of clinicians to take an integrated approach to diagnostics and to train specialists in integrated diagnostics. Davidson added that much could be learned from health systems that are implementing and scaling integrating diagnostics. Some of the lessons shared by speakers included facilitating structured reporting to integrate data across diagnostic specialties, ensuring feedback loops to assess diagnostic performance, and ensuring that a primary focus of integrated diagnostics is patient-centered care.

Improving Evidence Generation

Much of the discussion on improving evidence generation for integrated diagnostics centered on the opportunities to use EHR data and capabilities to support the evaluation of integrated diagnostics, Levy said, adding that this includes working with EHR vendors to advance and disseminate use cases. Many speakers said that EHR data collected during routine care could be leveraged for pragmatic studies and suggested being more intentional about collecting EHR data for prospective clinical trials. Levy said that speakers highlighted the need for “significant investment in evidence generation for analytic validity, clinical validity, and clinical utility” to support regulatory review and implementation of integrated diagnostics in clinical practice. Although challenges persist, “the culture of data sharing for discovery remains strong,” Levy stressed. She concluded that “regulations, policies, and guidelines that recommend minimum structured documentation standards for diagnostic reporting influence documentation practices that enables evidence generation for integrated diagnostics.”

Enhancing Design and Use of Integrated Diagnostics

Jensen said one of the recurring topics of discussion around design and use of integrated diagnostics was the potential for AI algorithms to perpetuate and amplify biases in training datasets, with many speakers highlighting the importance of training and testing models on datasets that reflect the diversity of the population. This includes ensuring that patient enrollment in clinical trials of integrated diagnostics is diverse and inclusive, which also expands patient access to cutting-edge research, because these prospective trial

populations are likely to be used for future research, including AI algorithm development.

Jensen said another topic was the use of DMTs to provide guidance to treating clinicians on test ordering and interpreting test results. Jensen said it might be helpful to develop a DMT on specific clinical areas at the outset and expand capacity to additional areas, rather than attempting a comprehensive rollout. Jensen also suggested that research funders support development of DMTs.

Naik and Jason Slagle, research associate professor in the department of anesthesiology at Vanderbilt University Medical Center, highlighted additional design and use considerations that speakers discussed, including the need for investments in data engineering to scale the integration of data from across the diagnostic specialties. Slagle said that successful implementation of integrated diagnostics also hinges on trust, understanding, accountability, timing options, prioritization, annotation, and actionability. He added that these issues affect clinicians’ perceptions of integrated diagnostics and their willingness to use them. Naik pointed out that once implemented, ongoing testing should incorporate “the principles of high reliability organizations to ensure reliability and reduce unwanted variation.” He also noted the need for open-source protocols for algorithms and interinstitutional sharing and testing of protocols to support scalability and sustainability.

Many speakers emphasized the need to redesign EHRs to better meet clinician needs and facilitate integrated diagnostics, Slagle said. Some of the suggested changes included incorporating data visualization, multilevel modeling, and ML approaches for workflow and diagnostics, he summarized. Using principles of risk communication to support shared decision making was also discussed as an opportunity to improve patient care. Naik added that some speakers suggested a code of conduct for implementing AI-based integrated diagnostics and also leveraging innovative clinical trial designs to facilitate simultaneous testing of tools and implementation strategies. Finally, Slagle noted that more data scientists need to be recruited to the fields of cancer research and care.

Regulatory Oversight and Insurance Coverage

Jensen suggested that developers of integrated diagnostics meet with FDA leadership to discuss regulatory pathways to evaluate them. In regard to insurance coverage, he noted “there seems to be a reticence … in paying for integrated diagnostics as a stand-alone component of medical care” and suggested that moving toward bundled payments for oncology care might partially address this issue.

user-centered design should address the needs of both the clinician and the patient as users. Karlan also emphasized the need to engage those users when developing new AI-based integrated diagnostics. Winn highlighted the difference between useful and usable. Something that is useful is “fit for a purpose” but not necessarily “fit to be used.” In the current context, data can be useful but not necessarily usable by different end users. The usefulness, and usability, of discoveries should be considered in parallel, he said, emphasizing that a critical disconnect in health is that there is a “lot of discovering, but that doesn’t cross the bridge to implementation,” and as a result, not everyone benefits from the advances. “That’s why I really think about the useful discovery, usefulness, and at the end, the usability,” he said.

REFERENCES