of Clinical Pharmacology and Office of Translational Science in the Center of Drug Evaluation and Research at the Food and Drug Administration (FDA). “Given the complexity of CNS disorders,” he said, “the use of multimodal biomarkers may provide better assistance for future new drug development.” He provided two examples of how multimodal biomarkers could be used to achieve drug development goals. In one case, a sponsor failed to show efficacy of a drug in a Phase 2 trial for patients with bipolar depression. To move to Phase 3, they proposed a machine learning approach in which they identified patients whose symptoms are characterized by greater similarity to patients in the target population; patients with high anomaly scores would be excluded. The FDA considered this proposal acceptable, but also suggested that one Phase 3 trial include the anomalous patients to improve the explain-ability of the model, better understand the generalizability of the results, and ensure that the targeted patient population can be appropriately defined in the program. In the second example Zhu provided, a FDA research project hoped to improve efficacy findings in patients with schizophrenia by using a machine learning approach to identify patients with biomarkers that predicted good response to placebo; these patients were excluded from the trial, which improved detection of the treatment effect (Figure 5-1).

There are critical methodological considerations for using machine learning approaches to evaluate patients based on their biomarker profiles. “The most important issue is how do we find a reliable data source to generate useful information,” said Zhu. He emphasized the need for validated approaches and consistency in measurements, endpoints, and patient recruitment. Data standards would also be important when pooling data

from multiple sources. When developing models of patients, it is important to make sure that the data supports the assumptions, that the model is fit for its purpose, and that bias is not introduced. To do so, it will be important that training data for machine learning and artificial intelligence models is diverse enough to capture the diversity of patient population. All these models will need to ensure that they are transparent, interpretable, and explainable. It will be important to build up best practices in this field.

Industry Perspective

In psychiatry, average effect sizes for drug efficacy are small, but there is a critical difference between treatments that have small effects on all patients and those for which a small minority of patients respond well, said Amit Etkin, founder and CEO of Alto Neuroscience. Biomarkers can be identified using data from electroencephalogram (EEG) (which is more scalable than magnetic resonance imaging [MRI]), behavior, and wearable devices, but it is critical, from an industry perspective, to ensure that the data are valid and replicable. Etkin provided an example of a study that used a machine learning algorithm trained on EEG data and other measures to produce a score that predicted responses to open label selective serotonin reuptake inhibitor treatment (Figure 5-2). Once the measure was identified, it was validated using a set of data that was held out of the initial analysis, replicated by applying the same algorithm to older, similar datasets, and evaluated for its test-retest reliability, which Etkin reported as up to 85

percent. Although they gathered many different modalities to feed into the algorithm, Etkin cautioned that it is best to use as few modalities as possible: “We aim for simplicity and interpretability in our models.”

Etkin also described an approach to developing a novel drug for patients with depression who show poor cognition. Because hippocampal plasticity plays a role in both cognition and mood, a drug candidate was developed that improved hippocampal plasticity in animal models. In Phase 2 trials in humans, it failed to show improvement in an all-comer trial¹ of patients with depression. However, using biomarkers for patients who show poor cognition, they were able to show better efficacy in those patients; the results from an initial small cohort of patients was later replicated with a much larger, prospective study. This type of approach is being used for development of several promising treatments. Etkin believes that the field can make significant progress by identifying patients with specific biomarkers and validating those biomarkers.

METHODS FOR VALIDATION AND STANDARDIZATION

Throughout the workshop, several workshop participants emphasized the importance of validation and standardization for multimodal biomarkers to be useful for clinical trials or for clinical use. The process can be particularly challenging in the context of CNS disorders, even for unimodal biomarkers or measures. Workshop discussions highlighted two areas, including development of measures of speech and standardization of brain imaging data, where progress has been made.

Validating Measures of Speech

Speech is a useful clinical measure because it is a complex cognitive and motor task, according to Visar Berisha, an associate professor at the College of Engineering and the College of Health Solutions at Arizona State University. Speech also serves as a broader indicator of function that can be useful for tracking disease and response to treatment, for example, by measuring social interaction in people with schizophrenia and communication among those with Parkinson’s disease (PD). Changes or abnormalities in speech may serve as an early indicator of disease (e.g., for Alzheimer’s disease [AD] or PD), but it is difficult to determine where the clinical information is in this “very high-density, high-velocity data stream” of up to thousands of samples per second. The current approach to finding meaningful signals (e.g., in a comparison of speech in healthy and AD patients) is to use an

___________________

¹ An all-comer trial design applies a wide inclusion criteria or very few (or no) exclusion criteria (Mandrekar and Sargent, 2011).

iterative, data-driven, supervised machine learning approach to extract the features of speech that best predict disease. However, Berisha cautioned that these purely data-driven solutions run into a variety of challenges (Arbabshirani et al., 2017; Berisha and Liss, 2022) and are unlikely to generalize; these studies suffer from overfitting of sparse data (Berisha et al., 2021; Vabalas et al., 2019). “More importantly,” Berisha said, “this sort of data-driven approach doesn’t allow us to make any natural connections to the existing knowledge base.”

Berisha described an alternative approach in which speech data are used to develop measurement models for constructs of clinical interest, which can be validated against determinations made by speech pathologists (e.g., articulation, prosody, and respiration) (Goldsack et al., 2020). These speech measurements can then be used to develop simple models that can help with assessments such as motor-speech, vital capacity, behavioral health, and cognitive screening. Five amyotrophic lateral sclerosis (ALS) patients participated in a study in which their speech collected at home was analyzed with a model that measured their articulatory precision (Stegmann et al., 2020, 2021); these data matched the clinical assessments, but with greater sample density and lower variability, said Berisha. These speech measurement tools are also being integrated into multimodal measures for other CNS disorders (Liss et al., 2010; Stegmann et al., 2022), including for early screening of PD alongside fine motor assessment tools (Kalia and Lang, 2015; Tanner, 2020).

Harmonizing Neuroimaging Data Across Clinical Sites

Harmonization of neuroimaging data is increasingly important because the number of studies is increasing exponentially, but many have small samples sizes that are generated at single clinical sites (Dennis et al., 2023). According to Emily Dennis, an assistant professor in the Department of Neurology at the University of Utah, this limits the generalizability and replicability of the findings. “Just because we have lots of data doesn’t mean that we can use them together effectively. We need to be sure that the data are comparable,” she said. In addition to differences in MRI machines and data processing introduced by different vendors, there are site-specific effects that make the data difficult to compare across sites (Bayer et al., 2022). When combining data, researchers would want to remove methodological differences, such as those related to the scanner platform, scan sequence, acquisition procedure, and postprocessing procedures. However, Dennis emphasized that it is important to retain information about the patient cohorts and their biological data.

One approach to harmonizing neuroimaging data is to do so prospectively so that different clinical sites harmonize parameters and procedures

before collecting data. This process could include the use of traveling human “phantoms,” people who are scanned on different scanners to ensure that each scanner provides the same data. Phantom objects (e.g., the CHIPS project) designed to mimic human tissues can also be used. Retrospective harmonization is more difficult, said Dennis, but it is important because there have been so many studies that could be used for new analyses (Bayer et al., 2022). Methods such as ComBat and ComBat-GAM harmonize imaging data by making linear or non-linear adjustments to site means and scaling site variance differences. Another consideration that will be driven by the end goal of the study is whether to harmonize imaging data as raw data (Cetin-Karayumak et al., 2019), which is computationally expensive, as derived imaging metrics (Onicas et al., 2022), or as summary metrics. Dennis believes that harmonization is important because “we should be able to leverage the vast amount of data that has been collected already.” Technical developments have improved harmonization efforts, she said, and opportunities are increasing as researchers become more willing to collaborate and share data.

RETHINKING CLINICAL TRIALS AND CLINICAL USE

Adam Ferguson, Director of Data Science in Brain and Spinal Injury at the Zuckerberg San Francisco General Hospital and a professor of neurological surgery in the Weill Institute for Neurosciences at the University of California, San Francisco, opened the discussion by noting that individual biomarkers are often noisy measures of a much more complex disease space; only by understanding cross-correlations among many biomarkers can the field begin to understand and treat disease with precision. He described Expert-Augmented Machine Learning (EAML), in which machine learning tools, such as those described by Etkin and Berisha, are used by clinicians in an interactive and collaborative way. EAML “will be critical to do real-world implementation of biomarkers,” he said, and user-centered design of decision support tools will be important to ensure that clinicians can best use them. Berisha pointed out that, although clinical guidance is important, it is still important to consider that, as more biomarkers are integrated into a decision support tool, the data required to thoroughly validate each one increases exponentially.

The integration of clinical diagnosis with multimodal biomarkers and the need for clinical validation of these biomarkers raises questions about methodology and variability in clinical assessment and diagnosis. One workshop participant noted that psychiatric and clinical scores are very difficult to harmonize, particularly across different clinical sites. Valentina Mantua wondered whether the field needs to rethink how to design clinical trials in the early phases of development for biomarker discovery, and

perhaps consider basket trials in which patients with different diagnoses are pooled based on the presence of biomarkers. Etkin said that diagnostic labels are still helpful, and that the science does not yet support the idea that you can “explain more variance using the biomarker than from the clinical indication.” Still, he said, this may be the direction that the field is heading.

Scalability of and accessibility of multimodal biomarkers will be critical, said Ferguson, to ensure health equity. “There’s a digital divide and there’s a health care disparity problem; you pair those together and you can anticipate that we’re going to worsen our health disparities.” Etkin suggested that assessment tools that can go home with the patient can help address this problem.