8

Closing Roundtable

In a brief final roundtable discussion, five participants—Heidi Williams (Stanford University), Jon Lorsch (National Institutes of Health), Alan Tomkins (National Science Foundation [NSF]), Caleb Watney (Institute for Progress), and Adam Jaffe (Brandeis University)—shared some of the major lessons they took away from the workshop.

Williams, who moderated the session, began by reiterating, as an example, the request by Diana Epstein (Office of Management and Budget) for a toolkit that would facilitate arranging for employees under the Intergovernmental Personnel Act (see Chapter 6). “Things like that that [provide] a to-do list for people who are trying to make things in the field work better,” she said.

Watney cited his interest in comments by Hoangmai Pham (Institute for Exceptional Care) on the launch of the Center for Medicare and Medicaid Innovation (CMMI) and how its structure and circumstances contributed to its successes and shortcomings. Such information “really matters both for setting up future organizations and for getting organizations right in the future.” Enlisting someone to do a deep oral history of the early stages of CMMI could provide a more complete picture.

Lorsch urged science and innovation policy researchers to incorporate some of the ethical thinking done in the world of clinical trials into their randomized controlled trials (RCTs). For example, ethical precepts could help ensure that researchers do no harm relative to a standard of care. “In the context of giving grants, issuing patents, or giving reimbursements for medical services,” Lorsch said, “the bar to doing an RCT should be extremely high because the standard of care produces some value.” Randomizing someone to a particular group could produce a worse outcome, “in which case the taxpayers are getting a worse service, and you are certainly going to hear about it.” If someone does not get a grant, a patent, or a reimbursement, people in the randomized group could attribute that outcome to being in the randomized group. He continued, “I would urge you to look at clinical trial ethics and try to superimpose that on what it is you think should be done in terms of government experiments.”

Lorsch also said that at some point, depending on how much money is involved, an experiment becomes a program, “and an old saying in government is that it’s very easy to start programs but very hard to end programs. Once you get them going, they take on a life of their own.” The instigators of experiments need some way to stop a program if the evidence shows that the program is not beneficial, he said.

Finally, Lorsch pointed out that “not every idea for an experiment is worth doing.” Just as thought needs to be devoted to what scientific experiments will give taxpayers returns on their investment, experiments in the policy sphere need to be considered in that light.

Citing the need for systematic inquiries, Tomkins called for more funding for research on the science of science. NSF has such a program, but other agencies should support this research as well. “It’s another area of scientific inquiry, and it’s one that is undervalued and underexplored,” said Tomkins. As a specific example, more research on public access to results and open science could help reveal the benefits and costs of this approach.

Jaffe said that he was struck by the statements that small amounts of money spent on evaluation and understanding can have very large impacts on the effectiveness of large expenditures. He also pointed to the value of collaborations that can enable work in agencies that do not have enough personnel to do much experimentation.

Finally, to close the workshop, Jaffe pointed out the major goal of figuring out “ways to help people move from trying new things to building a culture of systematic learning about what works better and what doesn’t work as well.” This requires a cultural change, which cannot happen quickly. “But I was always taught that the perfect shouldn’t be the enemy of the good,” he said. “We should keep trying to do better than we were doing before. The more that we specifically and systematically organize ourselves to be learning, the more we’ll learn.”