4

Protection of Research Integrity

The session on research integrity featured three speakers. Patricia Valdez, health science policy analyst at NIH and extramural research integrity officer in the NIH Office of Extramural Research (OER), spoke about NIH’s extramural research portfolio. Daniel Greenbaum, president of Health Effects Institute (HEI), discussed how to gain sponsor support while maintaining independence. Clive Green, executive director of bio-pharmaceuticals research and development at AstraZeneca, addressed the application of corporate ethical policies and governance processes with an emphasis on bioethics. The session concluded with a panel discussion

moderated by Aaron Kesselheim, professor of medicine at Harvard Medical School and faculty member in the Division of Pharmacoepidemiology and Pharmacoeconomics in the Department of Medicine at Brigham and Women’s Hospital, in which the three speakers were joined by Nicholas Chartres, director of science and policy for the Program on Reproductive Health and the Environment (PRHE) at UCSF, and Gary Ruskin, executive director and cofounder of U.S. Right to Know.

RESEARCH INTEGRITY IN EXTRAMURAL RESEARCH AT NIH¹

Valdez discussed NIH steps to reduce bias, in either selecting projects for funding or conducting research. She began by talking about rigor and transparency in grant applications; NIH defines rigor as “the strict application of the scientific method to ensure unbiased and well-controlled experimental design, methodology, analysis, interpretation, and reporting of results.”

In 2016, NIH implemented a policy called “enhancing reproducibility through rigor and transparency” to respond to concerns about the lack of reproducibility of preclinical data in publications.² It required grant applicants to describe their research in more detail and how they planned to produce rigorous research. The policy also allowed grant reviewers to judge applications based on the rigor of prior research and plans to address weaknesses in it; the rigor of the proposed research; the role of relevant biological variables, such as sex, on the proposed research; and authentication of key biological or chemical resources (see Figure 4-1). The new policy, Valdez explained, requires applicants to describe how they will achieve robust and unbiased results given the design of their experiments and the methods they plan to use. They also must provide information about the calculations and analyses they plan to conduct.

Avoiding Bias in Funding Decisions

Once a grant application comes into the NIH Center for Scientific Review (CSR), it is assigned to a study section, where the PI’s peers will evaluate it for scientific merit, Valdez explained. After the reviewers score the application, the grant goes to the relevant NIH Institute or Center’s advisory committee. This step, she noted, acts as an additional safeguard to ensure that it is making the correct funding decision. Next, NIH evaluates the proposal’s relevance and need before making the final funding decision and initiating a grant. NIH monitors the grant’s programmatic

___________________

¹ This section is based on the presentation of Patricia Valdez, National Institutes of Health.

² Additional information is available at https://grants.nih.gov/policy/reproducibility/index.htm (accessed February 3, 2023).

and business management performance using yearly progress reports that include information on how the applicant is ensuring robust and unbiased research.

Valdez said NIH holds dear several core values for peer review: expert assessment, transparency, impartiality, fairness, confidentiality, security, integrity, and efficiency. Impartiality refers to the management of COIs or the appearance of COIs, and reviewers must certify their COI statements both before and after a study section meets. NIH does not allow lobbyists to serve on study sections. In addition, NIH separates staff functions, so the person overseeing the conduct of research is not the same one who manages the review process. There is also an appeal process if an individual has concerns that the peer-review process was biased.

In terms of the integrity of the peer-review process, NIH staff receives training on how to handle allegations of research misconduct, said Valdez. Before each meeting, NIH instructs reviewers and council members to report an allegation to the designated federal officer, who then reports it to the assigned research integrity officer in the relevant institute or center. NIH may defer the application from review until the Department of Health and Human Services (HHS) Office of Research Integrity (ORI) has assessed the allegations. In addition to the peer-review process, allegations can come from outside NIH, such as the investigator’s institution.

Valdez noted that her office handles other integrity-related concerns, such as harassment, bullying, and discrimination, and issues with foreign interference and individuals trying to breach the integrity of the peer-review system. Allegations of grant fraud are sent to the NIH Office of Management Assessment for potential referral to the Office of the Inspector General.

Depending on the outcome of the allegation assessment by ORI and the pertinent institute or center, NIH may contact the investigator’s institution, remove an individual from serving on a peer-review committee, refer the allegation to the agency or office with oversight, or take administrative actions. In severe cases, NIH can take regulatory actions against institutions. Given a risk to a grant’s funds, the research, or human or animal participants, NIH can take interim actions, such as requiring additional supervision of the individual involved or certification of data, requesting a new PI, restricting funds, or even suspending or terminating an award. NIH may also refer a case or individual to the HHS Office of the Inspector General.

NIH’s financial COI policy is based on HHS regulation 42 CFR part 50 subpart F, which addresses promoting objectivity and establishes standards that provide a reasonable expectation that the design, conduct, or reporting of NIH research will be free from bias resulting from any investigators’ financial COIs. It requires investigators to report their financial

interests to their recipient institution, which determines whether a potential COI with the NIH work exists. If so, the institution will provide NIH with a management plan that NIH will either approve or reject.

Clinical Trial Registration and Reporting

Several years ago, said Valdez, NIH began requiring investigators to register all NIH-funded clinical trials at the ClinicalTrials.gov website within 21 days of enrolling the first participant and report a summary of the results within 1 year of the primary completion date.³ This is required regardless of the study phase or type of intervention, and it is subject to regulation. The policy requires a plan in the grant application that outlines compliance with the policy, which becomes part of the terms and conditions for an award. Valdez said that if an institution does not follow this plan, NIH can terminate an award and even take regulatory actions against an institution. In addition, all clinical trial consent forms must include a statement informing the potential participant about the information posted at ClinicalTrials.gov.

Valdez concluded with a brief discussion on NIH’s scientific data-sharing policy, which went into effect in January 2023. It took years to develop, and the idea behind it is that results, both positive and negative, produced using taxpayer funds should be available freely. “This is something that we thought long and hard about, and we are working with the extramural community to make sure that we do have proper plans in place and proper places for storing that data,” said Valdez. “We think this will be a very positive step going forward.”

When asked which of the methods of oversight she thinks are most relevant to protecting the independence of research, Valdez replied that the two-pronged peer-review process helps ensure that NIH is funding applications based on assessments by their peers. One thing her office sees more of is individuals citing NIH funding in their publications when it does not exist. They do this to get that paper indexed in PubMed, which provides legitimacy.

GAINING SPONSOR SUPPORT WHILE MAINTAINING SCIENTIFIC INDEPENDENCE⁴

HEI, said Daniel Greenbaum, was launched in 1980 after the Environmental Protection Agency (EPA) administrator and motor vehicle

___________________

³ Additional information is available at https://grants.nih.gov/policy/clinical-trials/reporting/index.htm (accessed February 3, 2023).

⁴ This section is based on the presentation of Daniel Greenbaum, Health Effects Institute.

industry chief executive officers agreed that an independent entity should produce the science on the health effects of vehicle emissions, which 1977 amendments to the Clean Air Act required. Greenbaum noted that while HEI’s sponsors provide input, along with scientific and environmental communities, into the priority topics HEI should address, sponsors have no role in selecting teams to conduct the research, overseeing its implementation, and reviewing its results before publication. Sponsors of HEI’s core air pollution programs come equally from EPA and the motor vehicle industry, and HEI has also built partnerships with WHO, Asian Development Bank, the U.S. Federal Highway Administration, and other industries.

HEI has an independent board that the EPA administrator and core industry sponsors approve. Without EPA and core industry sponsor approval, the board appoints two key committees: research and peer review. HEI’s commitment to transparency means it publishes all the results of the studies it funds, both positive and negative, along with a detailed commentary by the review committee. It also makes all its data accessible to others and does not take policy positions based on its work.

HEI has funded over 350 studies on a wide range of air pollutants, which have included research on exposure, toxicology, and epidemiology. HEI also conducts detailed reviews of the literature and has a global health program funded by foundations and multinational agencies conducting research and science communication in China, India, Africa, and other low- and middle-income countries. Recently, HEI started an energy program examining the potential exposures and health effects of unconventional oil and gas development (fracking).

Greenbaum explained that HEI produces and communicates trusted science through strategic planning and rigorous competition to produce quality policy-relevant science; being broadly transparent with study methods, results, and data; and not advocating for policies. The HEI strategic plan prepared every 5 years with input from its sponsors, the scientific community, environmental organizations, and others, provides a detailed plan for what should be happening to anticipate future policy and technology events and produce the science that can inform those events. He noted that the plan does evolve. For example, the current plan, produced in 2019 and 2020, did not include a research plan for air pollution and its relationship to COVID-19. Today, HEI has several such studies up and running.

The strategic plan informs a detailed process that Greenbaum said is similar in some ways to the NIH process Valdez described, involving soliciting grant applications, conducting a two-stage review, and overseeing the resulting research. Scientists populate the research committee, and HEI requires them to file COI disclosures and that no members of

the research committee work at EPA or the core industry funders. Once a project is selected for funding, it undergoes detailed oversight and quality assurance auditing and then must complete a comprehensive report of the findings that goes to the separate review committee. The review committee had no role in the study and implements an intensive peer-review process that includes outside reviewers. The review committee summarizes and communicates the findings through widely available HEI publications, social media, PubMed, and other channels.

Transparency, said Greenbaum, comes about in two ways. First, every PI is free to publish the results of their project in scientific journals, either before or after the internal review process, without HEI input. Investigators must also produce a comprehensive final report of all analyses. The report enables the review committee to understand both positive and negative findings and decide on appropriate ways to interpret the science and inform critical policy decisions. In addition, HEI expects investigators to make the data available publicly. Sometimes, HEI has funded the online publication of complete datasets that other investigators have used for additional studies.

Greenbaum explained that HEI does not take policy positions because of the concern that any subsequent research it might fund would be viewed as supporting that position. Because of this stance, HEI has been asked occasionally to reanalyze policy-relevant data independently. For example, in the late 1990s, EPA was considering a new standard for fine particulate air pollution based on the Harvard Six Cities Study results and an American Cancer Society study. EPA, industry, and Congress called for a reanalysis of the data, and the investigators from the two studies provided full access to their data. After conducting an independent reanalysis, HEI confirmed that the work was of the highest integrity and confirmed the original conclusions.

In another case, HEI funded a study to answer important questions about the cardiovascular effects of exposure to ozone. It involved nearly 90 older adults exposed to ambient and higher ozone levels using a carefully defined protocol (Frampton et al., 2017; Rich et al., 2020). It reported few effects on cardiovascular health but confirmed that ozone has respiratory effects in older adults, and EPA still uses these results to inform its decision making.

At the urging of its sponsors, HEI competitively selected three teams to examine the associations of exposure to deficient levels of fine particulate air pollution in 68 million Americans (Dominici et al., 2022), 8 million Canadians (Brauer et al., 2022), and 25 million Europeans (Brunekreef et al., 2021). These three studies provided evidence of associations with adverse health effects, said Greenbaum. He added that results are at the center of the decision-making process in the United States and Europe

around exposure to the setting of ambient air quality standards for fine particulate matter.

Greenbaum briefly mentioned that in the early 1980s, Congress funded the HEI asbestos research program. Schools were removing asbestos, with controversy about whether it should be removed from commercial and residential buildings. An independent expert panel reviewed the science and recommended strong steps to minimize asbestos exposure (HEI, 1991).

Greenbaum said that controversy and distrust between major parties helped to create HEI, a public–private leveraged partnership for research investment. HEI was designed for maximum impartiality and credibility, with sponsor input into priorities but no involvement in study selection, oversight, or review. HEI’s activities are guided by a carefully drawn, responsive strategic research plan renewed every 5 years as an opportunity to refresh and refocus its research. This approach has resulted in nearly 400 research studies, reanalyses, and systematic literature reviews widely cited in the literature and policy and regulatory deliberations.

Greenbaum noted occasions where regulators have quoted part but not all of a study’s conclusions and industry advocates will select the conclusions that are most favorable to them but leave out the accompanying caveats and comments. When that happens, HEI writes to the parties to clarify what it is and is not saying.

APPLICATION OF CORPORATE ETHICAL POLICIES AND GOVERNANCE PROCESSES, WITH A FOCUS ON BIOETHICS⁵

The focus on research integrity at AstraZeneca, said Green, is through the lens of bioethics, which is the practical application of ethics to a range of issues that arise from the study and practice of biological and medical science. Bioethics, he continued, is where AstraZeneca aims to build trust by demonstrating integrity, transparency, and fair treatment in everything the company does. Ethics and transparency, he added, is one of the three pillars of sustainability for the company, along with access to health care and environmental protection.

Bioethics at AstraZeneca is governed under the company’s global bioethics policy that covers a range of subjects, including clinical research integrity. Green noted that as a pharmaceutical company, AstraZeneca funds and conducts scientific research, and this policy covers sponsor influence in research that spans clinical and drug discovery activities. Some of the key principles in the bioethics policy regarding clinical research include the following:

___________________

⁵ This section is based on the presentation of Clive Green, AstraZeneca U.K.

• Maintain a portfolio of research and development projects designed to deliver effective, safe, differentiated medicines and address unmet patient needs.
• Conduct clinical studies per all regulatory requirements and recognized international quality and safety standards in all countries where the company operates.
• Make information publicly available about the registration and results of the company’s clinical trials for all products in all phases.
• Give those participating in the company’s clinical studies full, truthful, and understandable information and asking for their consent to be part of the study.
• Conduct preclinical studies to ensure that all safety aspects have been evaluated and that assessing potential risks and benefits justify testing a drug in the clinical setting.

Green said the company’s policy on transparency is to share its approach on its external website, share bioethics content through the company’s sustainability report, and participate in annual external audits. It also includes registering its clinical and observational studies, posting study results on disclosure websites, making a good faith effort to publish results in peer-reviewed journals in a timely manner, and providing transparency in bioethics actions following applicable legislation, regulations, standards, and guidelines.

To ensure compliance with this policy, Green said that bioethics responsibilities underpin the company’s overarching science policy, which is one of four pillars of the company’s code of ethics that defines its values and behaviors at work. Every employee, he said, participates in mandatory annual training on the code of ethics and research integrity and relevant governance policies and procedures specific to their role. Employees can report any concerns to a manager, human resources, legal department, compliance representative or an independent third-party group acting on the company’s behalf.

The company also has an internal bioethics advisory group that advises on implanting global standards, comprises individuals from a wide range of topics, and is supported by leaders from the company’s legal, compliance, and corporate affairs departments. It exists to provide advice, support, and guidance to scientists, project teams, and company leaders on bioethical issues, and it engages in horizon scanning to see where scientific, technical, and societal developments will prompt ethical challenges, said Green.

In addition, the company has an external advisory committee comprising leaders from academic and nonprofit organizations to provide access to expertise in pharmaceutical, research, and public health ethics

and law. The committee serves as a sounding board for proposals and gives advice and recommendations on ethical issues and societal perspectives. Both advisory groups sit outside the subject area governance structure to retain their impartiality. Green noted that the advisory groups do not get involved in the scientific detail of a proposed study unless a conflict with a company policy arises. A review of the decisions about the scientific details of a project is conducted through the company’s governance processes.

PANEL DISCUSSION: WHERE ARE THE POINTS OF INFLUENCE ON SCIENTIFIC RESEARCH?

Nicholas Chartres and Gary Ruskin reflected on the presentations. Chartres agreed that disclosing COIs is essential to identify and quantify the level of potential influence but does not protect the science from influence. As Bero noted, research synthesis and systematic review are also critical, and the challenge is identifying and quantifying potential influence when conducting systematic reviews that then inform guidelines or risk assessments used to develop recommendations for protecting public health. “If we do not have methods and approaches to identify COIs and industry sponsorship within those primary studies, and we do not attempt to quantify that level of influence, we may have an evidence base that can be skewed in favor of an industry’s product if that is the focus of the evaluation of the evidence,” said Chartres, adding that identifying and quantifying COIs in primary studies does not mean removing them from systematic reviews or meta-research but rather considering them in assessing a body of evidence.

Ruskin said that U.S. Right to Know focuses its research on the food and chemical industries, where a public–private partnership, such as HEI, is a nonstarter. “We know from our own investigative research on corporate documents that Coca-Cola and the ultraprocessed food industry have designed and executed an elaborate strategy to promote public–private partnerships as one part of their broader efforts to manage and control the public health discussion, to limit public policy options, to co-opt the public sector, and ultimately to defeat the public health community,” said Ruskin. “Based on the evidence that we’ve uncovered regarding these hidden motives of the food and chemical industry, such a public–private partnership around food or chemical research on policy is not likely to be good for public health.”

He applauded Green for his presentation on how the corporate world genuinely struggles with these real problems and questions. In his view, this type of corporate behavior should be encouraged, yet he also questioned the effectiveness of corporate self-regulation of research ethics

because corporations face no penalties for violating their guidelines. “Corporate codes of conduct can be weakened or abrogated at any moment,” said Ruskin, “and that is why solutions to these health research ethics problems will come through federal and state law and policy and not corporate self-regulation.”

Green remarked that not all industries are the same and that he has had the experience of being invited by other industries to talk about AstraZeneca’s model only to have them reject it because they wanted more involvement in research discussions. Greenbaum noted that one group missing from the day’s discussions is the scientific journal editorial community, given that most journals do not want to publish negative results, even for well-executed studies. Green agreed about that dearth of negative results, which will hinder future developments, such as powering AI models. He also noted the consequences for individuals who violate corporate codes of conduct and ethical guidelines.

Bero, speaking from her role as an editor for Cochrane, took exception to Greenbaum’s comment, stating that evidence from several studies shows that bias about negative results “is really submission bias.” Interviews with researchers suggests that some of this bias arises from a sponsor pressuring investigators not to submit negative results. She noted, too, that most medical journals now require publication of raw data and datasets in some open-access forum. However, she added, even though the raw data may be available to peer reviewers, many do not look at those data. “Publication is important,” said Bero, “but it is a last step in the research cycle, and we need to think about all the biases that can be introduced along the way.”

Redberg said that sponsors could play a role in encouraging or allowing the publication of all results, positive or negative. She explained that as a journal editor, she is interested in seeing studies that change practice, and negative results are just as important because they show what should not be done or does not work. Redberg agreed that data sharing is important, but not all journals require it. As an example, she noted how the Oxford Cholesterol Treatment Trialists Collaboration has a large body of evidence from statin trials, which have industry sponsors, but in 20 years, the collaboration has never made those results available because they are industry’s data. She called for industry to adopt policies similar to NIH’s on data release. Valdez noted that NIH’s new policy requiring grantees to share data will enable administrative actions against institutions or grant holders if they fail to publicly make their data available.

Kesselheim asked Chartres and Greenbaum if they have seen any trends in research sponsorship in their fields and to comment on the extent to which current protections for research integrity keep up with those trends. Chartres replied that his institution has industry documents

showing that since the early 1990s, the tobacco industry has been intentionally trying to undermine and suppress research around the harms of its products. Over the past 10–15 years, more documents have become available showing that multiple industries have used those same strategies. In terms of safeguards, Chartres believes that the scientific and research community needs to better understand that industry influence is happening, which is why he thinks this workshop is critical to informing the public and researchers about the extent of that bias and how it affects it public health and health care decisions.

Journals requiring COI statements are a step in the right direction, said Chartres, citing the Cochrane model regarding disclosure as one that most journals could use. Cochrane reviews cannot be commissioned or funded by any commercial sponsor that may have a vested interest in the findings of the review, and the first author must have no conflicts. He added that moving toward transparent databases is another important step and that the Physician Payments Sunshine Act, which increased transparency regarding pharmaceutical company payments, would be a good model to follow. Although some safeguards have been put in place, he said they are still inadequate given what the industry is doing to bias the research process.

Greenbaum said that a problem in environmental health research is that public funding has decreased, which raises the question of who the funders are. Fortunately, this field has a mechanism—HEI—for private funding to avoid some of the worst excesses of industry influence. But, at the same time, the limited public investment in research poses a challenge to the scientific community in some areas to continue to produce high-integrity, unbiased research.

Bero, commenting on the spectrum of funding in a particular area, cautioned that it is sometimes hard to understand how the sources have changed without doing meta-research. For example, when she looked at nutrition research, many researchers told her that the food industry funds everything, but it “is an important and influential funder, but it was not the major funder in the spaces we were looking at,” she explained. In contrast, it is hard to determine who funded a particular drug study because it was reported under the marketing budget and not the research and development budget. “You really need to dig into company reports to find out, and sometimes they are not transparent,” which makes it difficult to get a handle on trend data.

Tracey Woodruff, professor of obstetrics, gynecology, and reproductive sciences at UCSF, wondered if NIH, given the importance of research to the public’s health, might consider investing in the kind of meta-research that Bero discussed. Valdez responded that some NIH institutes and centers are funding more meta-research. She also noted that spotting

nondisclosure is challenging, particularly when foreign interference is involved.

Posing a question from the audience, Kesselheim asked Green if the experts on his company’s advisory committees are bound by nondisclosure agreements or can speak up about instances where the committees’ recommendations are not followed. Green cited an independent avenue for raising concerns in a safe and trusted environment. Ruskin noted that the life of a whistleblower is hard, though the qui tam process can be somewhat friendly. “We need to do everything we can to make whistle-blowing something that is accepted and supported in our health science culture,” said Ruskin.

Kesselheim asked the panelists for their thoughts on whether protections are sufficient to create public trust in institutions and research findings, and, if so, how best to communicate that so people can understand whom and what information to trust. Valdez replied that public trust is important; without it, people will not participate in clinical trials, for example. This is a difficult problem, she said, because of the disinformation issue, so building trust will depend on science education and ensuring that people are aware of what type of research and when they can trust.

Ruskin commented that the current situation is courting a crisis of confidence in the nation’s health institutions, given how heavily rooted they are in an evidence base that “may well not be trustworthy because so much of it may well be tainted by corporate influence.” Some estimates, he said, attribute nearly 60 percent of medical research funding to industry, and much of that research will be biased toward product defense and overstating product benefit. “Without a clean and uncorrupted evidence base, many people are just not going to trust or listen to scientific studies, public health leaders, medical institutions, regulatory agencies, and other health bodies,” said Ruskin. He cited a 2022 Pew Research poll showing that only 29 percent of U.S. adults say they have great confidence in medical scientists and scientists, in general, to act in the public’s best interest (Kennedy et al., 2022). Greenbaum added that transparency is critical to public trust.

Bero noted that preliminary data from a general public survey on COIs show that the researchers could not assess how it affects people’s trust because they did not understand the COI statements. The bottom line, she said, is that disclosure will not help with trust, so it is necessary to build people’s trust in the science and how it is conducted. Being transparent and making research understandable to a lay audience will be the key, which is why she is a fan of evidence synthesis. She noted that people trust Cochrane reviews and rate them highly because impartial experts vet their information.

Kesselheim asked the panelists for their ideas about the best opportunities to promote research integrity. Ruskin replied that the most important thing is for the message to come from the top: the president, Congress, governors, and state legislatures. “We have to tell the truth to the American people, that our current health evidence base may well not be that reliable and that as a matter of federal and state policy, we are going to do better and build a health evidence base that people can trust and believe in,” said Ruskin. “We cannot allow corporate [public relations] and product defense to pass as science anymore.” Valdez echoed that, adding that the “top” also includes the leadership of institutions because they can set institutional policies. She suggested that leaders, and not just trainees, should receive instruction on the responsible conduct of research.

Greenbaum commented that the more transparency in research funding, decision mechanisms, and oversight, the more the public is likely to trust it. He noted that unlike Green’s company, some companies do not want to be transparent about who is on their panels and advisory committees. “Figuring out how to improve transparency and make it much more consistently applied will be important,” said Greenbaum.

Bero agreed with these points about transparency but said the low-hanging fruit is developing structural mechanisms for ensuring independence from the sponsor, something that transparency does not guarantee. “We need to make sure that the research is under the control throughout the entire research cycle by independent investigators and funders,” said Bero.