3

Hurdles for AI for Scientific Discovery

TECHNICAL CHALLENGES TO AI PLAYING A SIGNIFICANT ROLE IN SCIENTIFIC DISCOVERY

The third panel, moderated by Missy Cummings (George Mason University), examined some of the limitations facing those who would use artificial intelligence (AI) to conduct independent scientific discovery and what might be done to overcome those limitations. This session focused on the current issues with AI that pose hurdles to achieving autonomous scientific research.

Cummings began by saying that the session was really “the curmudgeons’ session,” explaining that she and others in the session have reputations as curmudgeons, or people who raise doubts about technology. For instance, she pointed to issues with the convolutional neural nets used in self-driving cars—which have had many crashes and other failures—and said that large language models, which are being touted for use in scientific discovery, are built on the same technology.

How AI Will and Won’t Change Science

Gary Marcus (New York University), who spoke via a prerecorded message, was the first presenter. Referring to some predictions of how AI will change the world, Marcus said that it may, but right now it does not work as well as many people would like to believe. Illustrating his point with some examples, Marcus said that when today’s AI models are asked certain

questions, they often provide answers that contain nonsense and logical fallacies; indeed, AI is generally unable to make logical inferences of the sort that most people make with little effort (e.g., recognizing that if Tom Cruise’s mother is Mary Lee Pfeiffer, then Tom Cruise is Mary Lee Pfeiffer’s son). So, at this time, AI models cannot be expected to come up with scientific discoveries that rely on inference. Furthermore, he continued, many of the most impressive claims about what AI can do in the area of science have been found to be either wrong or dependent on getting the right sort of prompt from the people asking the question.

In closing, Marcus said that scientists can and should use AI, but they should not expect magic. Dedicated, narrowly focused AI models such as AlphaFold are terrific, he said, as long as users understand the limits of the tools. But expecting generative AI to invent new science—or, worse, to obviate the need for training scientists—is unrealistic.

AI for Scientific Discovery

Subbarao Kambhampati (Arizona State University) provided a general overview of the limitations of AI for scientific discovery. He began by noting some current trends in AI technology. It has been shifting, for instance, from explicit-knowledge tasks (whose rules can be articulated, such as chess) to tacit-knowledge tasks (whose rules cannot be articulated, such as vision) and from reasoning from specifications to learning from data. AI has also been moving from deep and narrow (protein folding) to broad and shallow (ChatGPT) and from discriminative classification (“Is this a dog?”) to generative imagination (“Draw a dog.”).

With deep and narrow systems, he continued, the main issue is human–machine interaction and, in particular, providing explanations. How, for example, can machines provide some sort of explanation to humans for their choices, and how can humans provide advice to a machine on its tasks? By contrast, with broad and shallow systems, the main caution should be machines’ lack of reasoning abilities, Kambhampati said. Large language models and other types of AI are great idea generators, but there is no reason to believe that their ideas are sound. So, it is best to use them to devise a large number of ideas for a given problem and then have human experts cull the ideas to see which are useful. Similarly, large language models cannot be trusted to create workable plans, but they can generate a large number of possible plans that can then be analyzed by an LPG (local search for planning graphs) model to zero in on the optimal plans.

The bottom line, he said, is that AI will certainly be helpful in scientific discovery, but it will most likely be as an assistant to human scientists rather than as an automated scientist.

Best Ways to Use AI

Chitra Sivanandam (Rohirrim) offered her thoughts on the best ways to use AI in scientific discovery. After reviewing the scientific method—from stating the problem to forming a hypothesis and then testing the hypothesis—she listed some common pitfalls in applying the approach. These include forming poor hypotheses, making invalid assumptions, being biased in design and evaluation, and making poor conclusions; each of these, she said, can be exacerbated by generative AI, especially if it is used to confirm underlying biases. On the positive side, she continued, generative AI provides an opportunity to investigate what is plausible by generating a large number of possibilities in dealing with a question or problem.

Those who use AI need to keep in mind the many gaps it contains today. First, there are many gaps and biases in the data and in the algorithms used to work with the data. For example, today’s large language models almost all rely on English-based data, which ignores a large amount of data available in the world. Biases also arise from gaps in prior research, since AI cannot use information that is not available.

The best uses for generative AI, she said, include conducting research and development of multiple hypotheses, continuously testing and evolving assumptions, seeking alternatives, identifying potential gaps or opportunities to evolve research, and summarizing conclusions. AI should not be relied on for designing experiments (which should always have humans in the loop to mitigate bias), determining conclusions or causation, validating math or science, corroborating conclusions based on references, or evaluating the validity of research.

Discussion

In the discussion that followed the presentations, Cummings summarized the panelists’ thoughts by saying that AI can be an important tool for human scientists, but it also needs to be supervised by human scientists. In response to a question from a member of the audience, Kambhampati said that large language models are very useful in generating written things such as emails or proposals, where a human can verify that the information is

correct, but a weakness appears when the model includes information that the human is not familiar with because these models cannot be trusted to validate their information.

Hod Lipson (Columbia University) asked the panelists if they believed the sorts of issues with AI that they had identified would always be a problem or whether advances in AI would solve those problems. Cummings answered by saying it is important to remember that while AI might get better, it is not because it is learning. AI is not thinking or learning, so some problems will always remain, even if they are not the same problems. Sivanandam said that while today’s large language models are mostly useful for helping scientists see problems in new ways, that might change in the future if machines learn to communicate and collaborate.

SOCIAL, LEGAL, AND ETHICAL CHALLENGES FACING THE USE OF AI FOR SCIENTIFIC DISCOVERY

The fourth session, moderated by workshop chair Bradley Malin (Vanderbilt University), addressed questions about the ethical, legal, and social implications of autonomous AI and the extent to which these can enable or hinder the development of AI. The importance of these issues is illustrated by how machine learning methods have been found to embed societal biases, such as systemic discrimination, into their outputs. As AI evolves and becomes more powerful and sophisticated, it will certainly reflect a variety of such biases. The presenters on the panel offered their assessments of what to expect and what will need to be done to make AI as fair as possible in its implications for various members and segments of society.

To frame the panel’s discussion, Malin posed two questions whose answers will shape the social, legal, and ethical repercussions of AI. First, he asked, “Machine learning needs data . . . but what data?” To illustrate the issues related to the choice of data, he showed a graph from an article in Nature that showed the racial backgrounds in published gene association studies (Devaney, 2019). Approximately 78 percent of all the individuals included in these studies were of White European ancestry, while only 2 percent were of African ancestry and 1 percent were Hispanic or Latin American. Thus, the results from these studies were far more likely to apply to White subjects than to Black or Hispanic ones. Explaining why that is important, he described a second study (Bentley et al., 2020) that showed that polygenic risk scores—that is, calculated risks of disease based on

multiple genetic variants—were far less accurate in those of African ancestry than in those of European ancestry, meaning that the bias in data collection had real-world implications for the health care of people in different populations.

Next, Malin asked a related question: “We need human scientists to set AI in motion . . . but who are the scientists?” As with the data available for use in AI, the scientists involved in AI are disproportionately White or Asian, and this lack of diversity among the developers of AI could lead to serious harm to individuals in underrepresented minority groups. As AI advances, it will be important to diversify both the data it uses and the scientists who develop it.

Legal and Social Issues Relating to AI

I. Glenn Cohen (Harvard Law School) began his presentation on the legal and social challenges facing AI for scientific discovery by identifying five points. The first point was that while “move fast and break things” may be a good motto for Silicon Valley and private industry, it is a terrible model for social effects and society. Unlike the sale of pharmaceutical products, there is no general U.S. agency overseeing all AI products to make sure that they will not cause problems for users or society. For instance, despite the disruptive changes expected with ChatGPT and similar products, there was no review or oversight of the products.

Second, Cohen said, people should have a legal right to know when they are dealing with AI. For example, a person providing personal information in an online chat on a corporate website should know if the “person” on the other end is actually AI.

Third, it will be important to restructure incentives so that the benefits of AI are distributed equally across society rather than concentrated among a small percentage of people who are already privileged. Today, for instance, many datasets used to train AI are biased toward certain types of people—White men, people in upper and upper-middle socioeconomic classes, and so forth—which means that the performance of the AI is likely to be uneven among different groups.

Cohen’s fourth point was that privacy is at risk, but most of the threat of AI has to do with inferences. Experts have given a great deal of thought to how to protect individuals’ information in a digital age, but AI will change the equation by making it possible to infer things about people even without having access to restricted data.

Finally, it will be extremely important to resolve issues related to intellectual property in the coming age of AI. In particular, discoveries made with AI may be much harder to patent and thus much harder to commercialize.

The Importance of Collaboration and Cooperation

Vukosi Marivate (University of Pretoria) offered his thoughts on a variety of issues related to AI, mostly on the importance of bringing together a large number of people to influence its development. In 2016, the Turing Award winner Geoffrey Hinton predicted that within 5 years deep learning would perform better than radiologists at interpreting medical images, but that never happened. A recent study, for instance, examined hundreds of efforts to diagnose COVID-19 with machine learning and concluded that none were reliable (Roberts et al., 2021). The problem, Marivate said, is that these efforts were generally carried out by AI and machine learning experts working with a lot of data but not collaborating with experts in the field of radiology. This situation offers some lessons, he continued, including that one cannot put all the responsibilities on the AI scientist or developer and that AI and data science practitioners must humble themselves to the subject-matter experts. Furthermore, he added, society’s expectations should be included in determining the limits of these technologies.

One way of advancing, Marivate said, is to provide “sandboxes” where different groups of people can get together to learn what works and what does not. Marivate highlighted the development of African large language models as one of the areas that African machine learning experts are collaborating on, due to the large number of African languages. He ended his talk by speaking about efforts on the African continent to bring people together to learn from each other and advance the field of machine learning and AI in Africa. “No one is coming to save us,” he said. “We are the ones we have been waiting for.”

Maintaining the Integrity of and Trust in Science

The third presenter, Deborah Johnson (University of Virginia), spoke about concerns she has about the effects of AI on the scientific enterprise and, particularly, public trust in science. She began by talking about the challenge of maintaining the integrity of scientific research when AI becomes embedded in the process. Science is a sociotechnical system, she

said, in which AI will affect and play a role in many, if not all, aspects of the system. Some of these parts of the system that may be affected by AI include trainings, incentives, and the peer review and publication systems. “All of those things are likely to make use of AI but also be affected by AI,” she said. “If we’re going to talk about AI in scientific research, we have to talk about that whole system.”

Johnson also cautioned about the use of the word “autonomy” in talking about AI in science. Not only is autonomy a very blurry concept, she said, but AI is not independent of human activity, interests, or values. Furthermore, thinking of AI models as autonomous can lead to major issues of accountability and responsibility. Accountability is usually associated with autonomy, she noted, and entities that are not autonomous cannot be held responsible for what happens. But if AI is conceived of as being autonomous, that can lead to a situation in which scientists can deflect accountability from themselves for what happens.

Another issue, Johnson said, is that the use of AI could lead to an increase in mistrust of science. Two major public concerns that can lead to a loss of trust in science are safety and bias, and the use of AI could amplify both of these concerns. This is exacerbated by the “black box” nature of AI outcomes. Why trust science, Johnson asked, when scientists do not understand how AI arrives at its outcomes?

Finally, Johnson said, she is concerned that the use of AI may lead to scientists becoming de-skilled, just as the growth in the use of calculators led to a decrease in mathematical skills among some students.

Discussion

Malin opened the discussion period by asking the panel to contribute thoughts on how to increase trust as AI becomes a more important part of science. Standards are necessary, Johnson said; without standards, one has no way to measure whether something meets those standards and thus is trustworthy. Cohen pointed to transparency. Unfortunately, many algorithms cannot be protected by patents and thus are kept secret to protect intellectual property (trade secrecy), which undermines transparency.

Another question focused on the energy costs of running computers for AI. It takes a great deal of energy to analyze all of the data, which adds to the problem of a warming planet. Cohen said that it is a familiar issue in economics—the tragedy of the commons, where the benefits of use are localized and individualized and the costs of the use are spread widely—and

added that addressing this issue will require increasing public consciousness of the costs.

Patrick Riley (Relay Therapeutics) asked what can be done to make sure that the benefits of AI are spread evenly among scientists and people everywhere. Johnson answered that it will be important to look for various ways to incentivize and disincentivize different choices to make the spread of the benefits of AI more equitable. Cohen said that choices about the licensure and pricing of AI have a major effect on the spread of its benefits.

IMPROVEMENTS IN AUTOMATED EXPERIMENTATION AND DATA COLLECTION THAT ARE NEEDED TO ENABLE AI FOR SCIENTIFIC DISCOVERY

Moderator Patrick Riley opened the first day’s final session by commenting that if AI is to successfully engage in scientific discovery, then it will be necessary for AI scientists to conduct experiments—to design, direct, and execute them. Indeed, one of the advantages of AI systems is that they are generally able to simultaneously manage more ongoing experiments than a human can. However, most automated experimentation systems of the sort that could be directed by AI systems have significant limitations. The session, Riley said, would be dedicated to discussing the current status of AI-driven physical experimentation along with the technology and opportunities that are on the horizon.

Needs in Materials Science

Benji Maruyama (Air Force Research Laboratory) discussed some of the needs for automated experimentation and data collection for AI in materials science research. He began by describing the Autonomous Research System (ARES) built at the Air Force Research Laboratory, which was the first closed-loop autonomous research robot for use in materials science. ARES taught itself how to grow carbon nanotubes at controlled rates by doing a loop of planning, experimentation, analysis, and iteration, doing about 100 experiments per day. One of the important gaps for this sort of AI research, he said, is software: “We don’t have the right software to make our machines talk to one another.” Hardware is also a problem, he continued. “Hardware is built for human interaction, not for autonomous or robotic interaction, so most of the things you try to automate are hard to do because they are not set up for robots.”

Another issue, he said, is that the focus on automation is too narrow. It is important to focus on human–machine teaming and the workflows that make sense. What should be done by humans, and what should be done by robots? Typically, robots do repetitive or dangerous sorts of work, but in the future, it will be useful for robots to carry out high-level cognitive work to augment the human decision-making process.

There are many data-related gaps and unknowns, he continued. What data should be collected? What metadata? How should miscalibration or corruption be dealt with? How should data be represented? How should experimental processes be represented?

Needs in Drug Discovery

Peter Madrid (Synfini, Inc.) spoke about the needs for AI research in the area of data-driven drug discovery. AI is playing a major role in drug discovery, he said, but it is not having equal effects across all phases of drug discovery. Madrid pointed to drug optimization as one particular area in which AI has not been as effective as desired. Speaking specifically of chemistry, he quoted an article in Nature that said that for chemists “the AI revolution has yet to arrive,” in large part because machine learning systems in chemistry do not have the accurate and accessible training data they need (Nature, 2023). In the design-plan-build cycle of drug optimization, Madrid said, there are multiple AI tools for design, but the other two areas are still done mostly by human researchers. However, Synfini has developed custom AI tools for use in the other two areas as well, which has expedited many of the steps in drug optimization.

Looking to the future, Madrid listed several major challenges in using AI for drug discovery. The primary bottleneck is validating AI designs with lab data, he said. Fully automated discovery will require tight integration of varied systems, covering both data and sample sharing. Building a fully automated lab is very costly, and there may be steps that are best left to human workers. Today, AI is still far from replacing drug discovery chemists, but it does have the power to make them much more productive.

Automating Discovery

The final presenter was Hod Lipson, who spoke about using AI for discovery. He began with general observations. First, he has observed that industry is moving faster to use AI in scientific discovery than academia

is, although he was not sure why that should be. He noted that there are different types of generative AI. His work, for instance, is based on evolution, which he characterized as being slower than other forms of generative AI but capable of “thinking outside the box.” And in every case, he said, experimental verification of AI findings is crucial.

In describing how AI can be used for discovery, Lipson began by speaking about earlier work he had done where AI was able to discern scientific laws from data on the movement of a double pendulum (Bongard and Lipson, 2007; Schmidt and Lipson, 2009). But that discovery relied on having specified the variables ahead of time. What if the variables were part of what needed to be discovered? So, Lipson has been working with AI to make discoveries about variables from videos of motion. From videos of a double pendulum, the AI model determined that only four variables were needed to describe the system, which is a correct result. In general, when the AI analyzed systems where the number of variables was known, it produced the correct answer. When the camera was trained on systems governed by an unknown number of variables, such as flames from an open fire, it would still provide an answer, and that value, Lipson said, offered insight into the system. Generally speaking, this AI model can provide information about the variables in a system—drawing graphs that illustrate them—but it does not say what the variables are. Indeed, he added, many important scientific discoveries had their roots in a scientist recognizing an important variable (acceleration in the case of Newton’s laws of motion, for instance).

Lipson offered four concluding thoughts: (1) AI can play many roles in science, including discovery, finding of relationships, and explanation; (2) AI has the potential to democratize scientific discovery, with more people being able to do science; (3) physical experimentation is the bottleneck, as AI-generated hypotheses must be tested; and (4) many scientific problems are too hard for humans but may be achievable by AI.

Discussion

Riley began the discussion session by asking what sorts of automated processes would not be feasible anytime soon. Madrid said that in chemistry, handling the necessary range of materials will take a while to perfect. Maruyama said it will be a long time before robots are as flexible as humans in terms of handling different sizes and shapes of objects. Another issue is being able to respond in real time to variations, which is something that trained humans can do but that automated robots have difficulty with.

More generally, humans can learn to respond to different feedback more easily than robots.

In response to a question about what one improvement in automated capabilities would be most useful, Madrid said that interpreting certain instruments, such as a mass spectrometer, would be a very useful capability for AI to have. Human scientists currently spend a lot of time on such interpretations.

Riley then asked what might be lost with a move toward more autonomous experimentation. Lipson noted that this would lead to a certain loss of experimental skills among human researchers. Madrid said that it could lead to a loss of creative insights and creative hypotheses.

This page intentionally left blank.