Advancing the Science of Patient Input in Medical Product R&D: Towards a Research Agenda

Proceedings of a Workshop—in Brief December 13, 2018

On May 9, 2018, the National Academies of Sciences, Engineering, and Medicine held a workshop titled Advancing the Science of Patient Input in Medical Product R&D: Towards a Research Agenda. This workshop, hosted by the Forum on Drug Discovery, Development, and Translation, focused on the science of patient input for medical product research and development (R&D), with consideration of downstream regulatory and post-market decision making. The objectives of the workshop were to consider the current state of the science for soliciting and incorporating patient input into medical product R&D, explore gaps in knowledge and other barriers that impede progress, and discuss a potential framework for a research agenda for addressing gaps and barriers that could help move the field forward.

Marilyn Metcalf from GlaxoSmithKline[1] told workshop participants the meeting would focus on three main topics regarding the science of patient input: (1) understanding the experience of patients with a disease or a medical condition; (2) gaining insights on patient perspectives and preferences regarding the benefits and risks of treatments; and (3) considering ways that patient input could further continuous improvement of clinical trial development. The goals for the workshop were to identify disease-specific approaches that could be applied more broadly across therapeutic areas, explore lessons learned from patient-reported outcomes (PROs) about developing and validating research methods, and consider early and clinical R&D applications for soliciting patient input that could help inform medical product decision making.

Cynthia Grossman from FasterCures[2] defined the science of patient input as “the development and use of systematic approaches and tools to collect, analyze, and apply patient input to the medical product R&D lifecycle and regulatory decision making.” Patient input can be a valuable source of information because patients are experts in their disease and the treatments they receive. “We know we need to do a better job of hearing directly from patients and their caregivers about the symptoms they experience, their experience with treatments, the expectations of benefit[s], and how they deal with uncertainty,” said Grossman. The enriched, patient-derived understanding of disease burden or natural history of disease may help sponsors develop products that may work better for patients. Grossman also noted that part of building the science of patient input may involve identifying and addressing knowledge gaps, such as workforce needs and systems for collecting and analyzing patient input.

Ahead of the workshop, workshop participants received a worksheet that laid out specific questions for consideration (discussion tool). The discussion tool, developed by the workshop planning committee, was designed to solicit contributions from workshop participants on the data needs, gaps and barriers, and possible research approaches relevant to the topic of discussion for each workshop session. The discussion tool was used to help guide breakout sessions, which were informal discussions convened after each plenary session. One workshop participant from each breakout session served as a designated scribe and provided a brief verbal summary of the discussions within that breakout session before the fully assembled workshop participants. A plenary discussion was then moderated by Mark Trusheim of Co-Bio Consulting and the Massachusetts Institute of Technology’s (MIT’s) Sloan School of Management. Trusheim reiterated that the goal of the workshop was to inform a research agenda that would help advance the science of patient input.

Figure 1 Example of the discussion tool.
SOURCE: Trusheim presentation, May 9, 2018.

Click to view larger in a new tab>>


To frame the workshop discussions and provide concrete examples of how patient input can be incorporated into medical product R&D, experts delivered a series of lightning round presentations focused on three topics of interest: (1) understanding the patient experience with a disease or a medical condition, (2) capturing patient perspectives and preferences on benefit–risk, and (3) incorporating patient input in clinical trial development and continuous improvement.

Topic 1: Understanding the Patient Experience with a Disease or a Medical Condition

In the first session, speakers described methods for capturing patients’ experiences. The workshop participants then separated into breakout sessions to discuss the research gaps and possible approaches for addressing those gaps before reconvening for a plenary discussion moderated by Trusheim.

Collecting Data Online

Lauren Bataille from The Michael J. Fox Foundation for Parkinson’s Research[3] (MJFF) discussed an online research study for Parkinson’s disease (PD) called Fox Insight[4] for which study participants complete virtual study visits and answer questions about their family and medical history, current medical conditions, quality of life, and activities of daily living every 90 days. The 90-day period was chosen as a retention tactic to keep study participants involved over the minimum 5-year study period. The online study allows a broader population of patients that better represents PD epidemiology to participate in research. It also supplements in-person studies with rigorous patient perspective data gathered over time using validated clinical assessment instruments adapted for online use. Bataille anticipates 40,000 study participants will be enrolled by the end of 2018. She added MJFF is testing a voice capture technology for use by individuals for whom PD has compromised their ability to easily use a computer or a smartphone.

Study participants have opportunities to contribute additional data, such as daily movement information collected through wearable devices or genetic information through a partnership with 23andMe.[5] Fox Insight can also administer surveys developed by outside collaborators. One such survey is collecting information on study participants’ most bothersome symptoms every 180 days. The first 5,000 responses enabled researchers to build a natural history of PD that shows how different symptoms become more important to study participants depending on where they are in the disease trajectory, explained Bataille.

Jennifer Liao from Evidation Health then discussed her company’s work with the Framingham Heart Study and MIT’s Computer Science and Artificial Intelligence Laboratory to identify a voice-based biomarker to detect cognitive decline,[6] which is a difficult task using PROs and questionnaires because cognitive decline occurs over extended periods. Evidation Health is developing this biomarker using advanced analytics to mine a longitudinal, multi-generational dataset of recorded voice sessions and associated medical records with diagnoses and physician notes. So far, the project identified the type of voice recorder that works best for the individuals who would enroll in a prospective study to test this biomarker and determined how to efficiently process the resulting voice data. Liao summarized by saying the Evidation Health team hopes that the high frequency of data collection enabled through the passive collection of voice data and the high fidelity of the data can support earlier detection of cognitive change as well as provide information on how the patient perceives his or her experiences of that decline.


To frame the workshop discussions and provide concrete examples of how patient input can be incorporated into medical product R&D, experts delivered a series of lightning round presentations focused on three topics of interest: (1) understanding the patient experience with a disease or a medical condition, (2) capturing patient perspectives and preferences on benefit–risk, and (3) incorporating patient input in clinical trial development and continuous improvement.

Topic 1: Understanding the Patient Experience with a Disease or a Medical Condition

In the first session, speakers described methods for capturing patients’ experiences. The workshop participants then separated into breakout sessions to discuss the research gaps and possible approaches for addressing those gaps before reconvening for a plenary discussion moderated by Trusheim.

Listening to Patients

Theresa Mullin from the U.S. Food and Drug Administration (FDA) said the agency receives compelling insights from patients at each of the patient-focused meetings[7] it held over the past 5 years. One insight was that development programs are not involving patients early enough in the translational phase of research. It is important to understand what aspects of disease burden and treatment matter most to patients and their families and how to measure them, what aspects of clinical trials can be better tailored to meet the needs and interests of potential clinical trial participants, how to better integrate PRO data or elicited patient preferences into assessments, and how best to communicate information to patients and prescribers, said Mullin. She also said the challenge is to industrialize the process of collecting patient input—to develop approaches for answering these questions that are not reinvented for every clinical trial and that are appropriate for all subgroups, including underrepresented populations and patients who have conditions that are not represented by well-organized, well-funded advocacy groups.

Mullin then listed four key topics that FDA plans to address regarding patient-generated data in a new set of forthcoming guidances:

  1. Collecting comprehensive patient community input on the burden of disease and the current therapy.
  2. Developing holistic sets of impacts, such as burden of disease and treatment, that are most important to patients.
  3. Identifying and developing good measures for the identified set of impacts to use in clinical trials.
  4. Incorporating measures into endpoints considered significantly robust for regulatory decision making.


Figure 2 Examples of when proposed FDA guidances on patient input could be applicable in medical product R&D. NOTE: Clinical Outcome Assessment (COA), Clinical Trial (CT), U.S Food and Drug Administration (FDA)
SOURCES: Mullin presentation, May 9, 2018; (accessed December 4, 2018).

Click to view larger in a new tab>>

These guidances will help clarify a development pathway for using powerful, qualitative information to set endpoints that could be used in a clinical trial and indicate the kinds of methods and approaches that would give FDA confidence in the data during the review of a regulatory submission.

Using Qualitative Methods and Real-World Data Collection Methods

Arthur Stone from the University of Southern California said the most common—and problematic—method for collecting self-report data is through retrospective questionnaires and interviews. He explained that the problem with retrospective questionnaires is that human memory cannot reliably recall details as accurately as researchers prefer. To eliminate retrospection and delve more deeply into patients’ lives, researchers developed real-time or near-real-time collection strategies using methods including daily diaries, day reconstruction, experience sampling, and ecological momentary assessment. Ecological momentary assessment focuses on a patient’s current state by sampling him or her randomly over periods ranging from 1 week to 1 month using techniques such as a brief, prompted questionnaire delivered by a smartphone.

Stone proposed switching from artificial cognitive interviews for real-time assessments to real-time cognitive interviews that speak directly with patients, which would provide a much richer, deeper understanding of patients’ experiences of a disease. He believes it is important to involve anthropologists, sociologists, and psychologists in the process of developing questionnaires for use with this approach. When asked if this approach would increase patient burden, Stone said a planned study will look at how many times a patient can be interviewed over how long of a time period before he or she feels burdened.

Topic 2: Capturing Patient Perspectives and Preferences on Benefit–Risk

The second session featured presentations on patient preferences and how patients perceive the benefits and risks associated with using particular medical products. The session also covered perceptions of benefits and risks during different stages of an individual’s disease progression and experiences and familiarity with different treatment options. Brett Hauber from RTI Health Solutions[8] noted that in the guidance on patient preference information[9] (PPI), FDA’s Center for Devices and Radiological Health (CDRH) defines PPI as “qualitative or quantitative assessments of the relative desirability or acceptability to patients of specified alternatives or choices among outcomes or other attributes that differ among alternative health interventions.” Following a breakout session, the workshop participants reconvened for a moderated plenary discussion.

Identifying and Assessing Methods for Eliciting Patient Preferences Across the Medical Product Development Lifecycle

Mats Hansson from Uppsala University described the PREFER[10] project, an ongoing project to understand what matters to patients, how it matters, and what matters most at different decision points in the medical product lifecycle. PREFER is a 5-year European collaboration funded by the Innovative Medicines Initiative, which is a public–private partnership aimed to speed up the development of better and safer medicines for patients and identify and assess methods for eliciting health-related patient preferences. PREFER aims to develop science-based recommendations for regulatory authorities, industry, and health technology assessment bodies on how, when, and in what context to integrate patient preference research in the medical product development cycle.


Figure 3 Stated goals of the Innovative Medicines Initiative’s PREFER project on patient preference research.
SOURCE: Hansson presentation, May 9, 2018.

Click to view larger in a new tab>>

One conclusion reached by PREFER—based on an exhaustive literature review, semi-structured interviews, and focus groups—is that there is no consensus on the definition and role of patient preferences or on how to conduct patient preference studies, said Hansson. The literature review identified 32 patient preference exploration and elicitation methods that researchers have used when developing medical products and devices. PREFER divides these methods into two categories: qualitative preference exploration methods and quantitative preference elicitation methods. Hansson added that patients expressed concerns about being informed, having a say in the development process, and understanding whether their views will inform regulatory decision making.

Hansson noted that from an ethical perspective, it is important for patients to have the possibility of being informed, but this can be a challenge methodologically. He added that patient preference is about knowledge and values, and values should be respected. “We should not try to force patients into valuing things that are not really what they would like,” said Hansson.

PREFER is conducting case studies in rheumatoid arthritis, neuromuscular disorders, and lung cancer. The rheumatoid arthritis study is examining what patients prefer when adding another drug to their current treatment and whether interactive information will affect those preferences.

Learning from Examples of Patient Preference Research Methods for Regulatory Decision Making

Hauber discussed two examples of how regulators have used patient preference research methods. In one study, FDA developed a stated preference survey to elicit benefit–risk patient preferences for multiple attributes of hypothetical obesity devices.[11] In 2015, FDA approved the first device therapy for weight loss since 2007, considering the results of this quantitative study as evidence that reasonable patients informed about potential risks would accept them in exchange for potential benefits commensurate with that provided by the novel device. A Genentech-sponsored study of patient preferences for subcutaneous or intravenous rituximab,[12] which can be used to treat blood cancers, used a crossover design that exposed patients to both modes of administration and then asked which they preferred. The results from this study were included in the patient experience section of the product label.





Figure 4 Two examples of patient preferences for medical devices being integrated into medical product development.
SOURCE: Hauber presentation, May 9, 2018.

Click to view larger in a new tab>>

The two different methods are for two different purposes, said Hauber. The FDA study used hypothetical scenarios and produced results as well as a decision aid tool that could be applied to multiple decisions, while the Genentech study used actual treatment exposure to inform a specific medical product decision. Hauber suggested that these examples indicate different methods may be suitable for different purposes and may depend on the needs of specific decision makers. He raised the question of how the decision maker may evaluate the information generated by a given method. “We need to step back and look at the broader landscape when we talk about patient preferences and say what are the needs and how can patient preferences and patient preference information begin to fulfill those needs,” said Hauber.

Considering Patient Preferences in Regulatory Decisions

Kathryn O’Callaghan from FDA’s CDRH highlighted a second regulatory use case for considering patient perspectives in weighing benefit and risk information, as well as two key resources FDA created to help move the field forward. As an example of how CDRH now uses PPI to assess risk tolerance, she discussed a case involving devices used for at-home renal dialysis. End-stage renal disease (ESRD) has been a focus area for FDA given the public health impact and unmet needs that could benefit from medical product innovation. FDA sought early engagement with the patient community and other stakeholders via a public workshop hosted by the Kidney Health Initiative in 2015.[13] Most ESRD patients (about 87 percent) receive in-clinic hemodialysis, which many expressed came with inconveniences due to the clinic visits three times per week. Home hemodialysis (HHD) could potentially alleviate many of these inconveniences, but comes with a benefit–risk tradeoff, including the potential for rare but serious adverse events and even death. There was a concern that without trained medical professionals to deal with those situations, HHD would not be acceptable, said O’Callaghan. FDA decided to clear the original HHD system for use with a care partner as a risk mitigation measure. However, this excluded patients without a care partner at home from using HHD. In reevaluating the initial decision based on feedback from patients, FDA considered information from a PPI study, which included existing HHD patients and used threshold technique questions to identify the risk tolerance threshold (i.e., at what levels of risk the patients would switch their preference from at-home, self-care HHD versus in-clinic hemodialysis).

One finding was that patients expressed a range or a heterogeneity of preferences showing the importance of not oversimplifying with assumptions that they all want the same thing. Another key finding was that a substantial portion of patients was willing to accept risk levels associated with at-home, self-care HHD. Considering the results of this study, FDA cleared a change in the indications for use to remove the care partner requirement.[14] This case study illustrates the potential value of PPI to inform benefit–risk considerations, including the appropriateness of risk mitigation measures, which may be important for both patient needs and regulatory decisions. O’Callaghan highlighted the importance of early engagement with FDA on the purpose and objective of the PPI study so FDA may provide input to proactively address issues that are specific and critical to enabling regulatory use of this information. To help with early planning discussions, FDA developed a framework to assist those interested in conducting PPI studies to support medical product development.



Figure 5 Potential uses of patient preference information at different stages of medical product development.
SOURCE: O’Callaghan presentation, May 9, 2018.

Click to view larger in a new tab>>

O’Callaghan noted the strength of evidence FDA needs to see from a patient preference study differs depending on the purpose and objectives of the study, so being able to specify those early is key to getting the feedback from FDA needed to help ensure the results will be relevant to the regulatory decision. She added that while there is substantial and growing literature on patient preference research, it is still early regarding its use for regulatory applications and that sponsors are only moderately willing at this point to bring this information to FDA.

Currently, there is limited system capacity to perform rigorous, high-quality PPI studies, said O’Callaghan. Given the limited research capacity, she favors focusing on several higher priority areas, with the “sweet spot” for such studies sitting at the intersection of preference-sensitive patient decisions; relevance to effectiveness, safety, and other important attributes; and where there is sufficient knowledge about patient perspectives on disease, existing options, and the characteristics of the proposed new option. O’Callaghan also identified a need for a roadmap for how to leverage expertise and experience from “neighbor sciences,” such as (1) workforce training from health economics and outcomes research fields, (2) qualitative understanding of the patient experience from the PROs field, and (3) how to assess real-world evidence for relevance, quality, heterogeneity, and generalizability.

Mary Tobin of the Alliance for Clinical Research Excellence and Safety raised the issue of how to best get information from patients that elicits what anthropologists call covert information, which is information people do not know to bring up unless asked in a particular manner. O’Callaghan said some qualitative work is being done at the earlier end of the development process to uncover covert information. Metcalf suggested the science of patient input field should start looking into collaborations with social scientists to study motivation, unintended consequences, covert information, and other pieces of information currently not captured clinically.

Topic 3: Incorporating Patient Input in Clinical Trial Development and Continuous Improvement

The final session focused on the role patient input could play in clinical trial development. The workshop participants again separated into breakout sessions for discussion following the presentations then reconvened for a final moderated plenary discussion.

Optimizing Clinical Trial Protocols with Patient Input

To develop a new framework for optimizing clinical trials, AstraZeneca ran an onsite study simulation involving 18 clinical trial participants who went through the rigors of a clinical trial from completing a mock informed consent procedure through a mock dosing visit, explained Lynn Hagger of AstraZeneca.[15] After consulting with the mock trial participants, AstraZeneca made 21 changes to its clinical trial protocol, such as providing trial participants with simple background materials in lay language, better educating trial staff about the study, making the trial setting more comfortable, and creating a Web portal where trial participants could examine their results. Trial participants wanted their time and schedule valued and they wanted to be reassured about confidentiality, said Hagger. The changes AstraZeneca made increased recruitment and retention in two subsequent trials and produced a net savings of $1.1 million.

Running a mock trial at actual clinical trial sites was expensive and required a great deal of effort for all involved. To address that problem, the company partnered with PatientsLikeMe (see next section) on more than a dozen studies to get patient input at the design stage. It also established a patient adviser partnership program that enrolls patients from different countries, socioeconomic status, and education levels. Using both an online community and patient advisers produces results similar to those from the onsite simulations for a fraction of the cost, effort, and time, said Hagger. This protocol optimization procedure is now standard for every AstraZeneca study.



Figure 6 Summary of the structure and lessons learned from onsite study simulations with clinical trial participants for feedback on study design and conduct.
NOTE: AstraZeneca (AZ), diagnosis (dx), Georgia (GA), Lupus Nephritis (LN), Pennsylvania (PA), Systemic Lupus Erythematosus (SLE)
SOURCE: Hagger presentation, May 9, 2018.

Click to view larger in a new tab>>

Although each trial and each disease area is different, trial participant sentiments fall into four categories, said Hagger. Trial participants want to be:

  1. empowered with information about the trial and how it will affect their disease status and treatment management;
  2. treated as individuals and have their needs, values, and preferences accounted for;
  3. provided with access to care and care coordination during their participation; and
  4. afforded continuity when transitioning out of the study in terms of the follow-up care they receive and transparency on outcomes.



Figure 7 Summary of clinical trial participant sentiments regarding trial experiences.
SOURCE: Hagger presentation, May 9, 2018.

Click to view larger in a new tab>>

Measuring Patient Experience to Improve Clinical Trials

Anuj Patel from PatientsLikeMe[16] discussed his organization’s TrialMark initiative to develop and test a standardized measure for trial participant experience in clinical trials. This initiative also aimed to create infrastructure to deploy the measure in clinical trials, collect the de-identified results in a single database, and develop analytics and tools to help companies improve trial participant experience. The PatientsLikeMe team interviewed a large number of patients with a variety of disorders in its online communities and identified more than 40 different domains of patient experience. Using this information, the team developed 3 modules of 12–20 questions each that trial participants answer at the first visit to a trial site, at a defined midpoint of the trial, and at their last visit to the site.


Figure 8 Goals of the PatientsLikeMe TrialMark iniative to develop standardized tools for measuring clinical trial participant experience.
SOURCE: Patel presentation, May 9, 2018.

Click to view larger in a new tab>>

PatientsLikeMe is now developing infrastructure to use this instrument in electronic form, and to collect and pool de-identified data from a trial to inform better clinical trial design moving forward, said Patel. He also said that over the next 12–18 months PatientsLikeMe hopes to provide insights, such as the types of initiatives that lead to better retention or shorter recruitment periods. Patel said a research question that needs addressing is whether a more concise instrument with fewer questions will have the same power and sensitivity as the current instrument. He added that the plan is for this instrument to be a free-to-use measure that any trial sponsor can adopt with the sole requirement of adding de-identified data to the TrialMark database.


Figure 9 Diagram of TrialMark measurement development, testing, and use.
SOURCE: Patel presentation, May 9, 2018.

Click to view larger in a new tab>>

Engaging Patients as Partners to Drive the Mission to End Breast Cancer

Joy Simha from the National Breast Cancer Coalition[17] (NBCC) described its 5-day intensive Leadership Education Advocacy Development workshop that teaches people who have been affected by breast cancer how to read papers, proposals, and think critically, which she said are actions applicable to any disease. In fact, Karlin Schroeder of the Parkinson’s Foundation noted that NBCC has provided insights and input into the Foundation’s training program. Simha said when the breast cancer landscape changes with new knowledge, the leadership project incorporates the changes into the program. Simha stated the most important outcome from the project is that it puts trained patient advocates in meaningful positions at decision-making tables, including in the peer-review process for proposals and grants. She said researchers have embraced NBCC advocates and include them from the beginning of projects. NBCC advocates who are brought into projects must be included in every research team meeting; otherwise, NBCC will leave a partnership.

NBCC learned through its work that change is hard and the potential for conflicting agendas highlights the importance of defining roles in a partnership, said Simha. One large barrier to progress has been that trials often stop even when it might be important to continue measuring patient outcomes. “We believe strongly that [trained] advocates at the table can help everybody stay focused,” said Simha in closing.


Figure 10 Summary of gaps and barriers experienced by trained National Breast Cancer Coalition advocates in the course of their work. 
SOURCE: Simha presentation, May 9, 2018.

Click to view larger in a new tab>>


Following each set of lightning round presentations, plenary discussions were then moderated by Trusheim. The following sections summarize the gaps and barriers to advancing the science of patient input, potential approaches to addressing those gaps and barriers, and potential stakeholders to engage in future efforts to advance the science of patient input as discussed by individual workshop participants.

Topic 1: Understanding the Patient Experience with a Disease or a Medical Condition

Identifying Gaps and Barriers

Individual workshop participants discussed the gaps in knowledge and the barriers to progress with regard to understanding patient experience with disease. Several workshop participants highlighted challenges related to data and data collection. Donna Cryer of the Global Liver Institute highlighted that discussions by workshop participants in her breakout session focused on the limitations associated with the use of prospective versus retrospective registries, which may not have been created specifically for research purposes. Jaye Bea Smalley of Celgene mentioned that in her breakout session, workshop participants discussed the lack of data on patient experience with a disease or a medical condition and patient input in electronic health records (EHRs). Relatedly, there seems to be a lack of data on patients in the transition between healthy and diagnosed with a disease. This stage represents a key piece of the patient journey and may be important for informing treatment, stated Richard Willke of the International Society for Pharmacoeconomics and Outcomes Research. Lana Skirboll of Sanofi emphasized a need to train a workforce beyond physicians that can support the effort to collect patient input data. Suzanne Schrandt of the Arthritis Foundation mentioned a corresponding need to establish a way to curate, share, and harness those lessons learned.

Another barrier to understanding a patient’s experience with disease raised by some workshop participants was the differential experience of patients resulting from disparities in health care delivery, which may effect how a given patient reports the natural history of his or her disease. Cryer said it is important to capture opinions from the “silent majority” of patients who are not connected to research studies by building relationships to engender trust and confidence among those often underserved patient populations.

Exploring Methods and Approaches

Several workshop participants shared ideas on current methods to capture patient experience with disease and approaches to improve them. Research to better understand what patients want should be prioritized, according to Kevin Fowler of the Kidney Health Initiative. Such research could be done in the precompetitive space, suggested Carol Meyer of Takeda Pharmaceuticals, and the impact of patient input on research could be studied and widely published, said Skirboll, Stone, and Hildy Dillon of the Cancer Support Community.

Cryer emphasized that it is important to recognize that “patient” is not a one-size-fits-all term and that patients with different levels of familiarity with research could be involved in different ways. Schroeder suggested that training programs could be developed for patients and their caregivers on how to engage in research. She also emphasized developing corresponding programs for researchers including patients in research. Many patient advocacy groups have existing models for these types of programs that could be industrialized. Input from trained and untrained patients could be balanced to address biases that might be introduced by training patients in the context and conduct of medical product research, said Tobin and John Wagner of Takeda Pharmaceuticals.

Collecting information on the natural history of disease was highlighted by some workshop participants as an approach to better understanding patient experience with disease. Cryer said that knowledge of disease natural history could help guide the development of biomarkers and inform business decisions and that methods for collecting natural history information include social listening, qualitative interviews, surveys, and existing data sources such as registries, epidemiology studies, claims, and EHRs. Precompetitive consortia aimed at collecting natural history data could develop methods for technology to aid in collecting and analyzing patient input, suggested Ronald Bartek of the Friedreich’s Ataxia Research Alliance.

Several workshop participants discussed the importance of respecting patients’ and their families’ or caregivers’ time throughout research activities. Mullin cautioned that “one thing we do not want to see is everybody rushing out to talk to patients” and observed that data repositories could be created and shared to avoid revisiting the same questions and increasing the burden on patients, families, and caregivers. It is important to engage patients as partners early in the medical product development process and ideally keep the same group involved throughout the entire development process, said Hagger. Hagger and Trusheim suggested “standing groups of patients” could serve as a national resource for precompetitive consortia.

Hover over or click any circle to pause text rotation. 

Considering Stakeholders to Engage

To gain further insight into the patient experience with disease, some workshop participants considered additional stakeholders that could have beneficial perspectives. Tobin and Metcalf suggested that because clinicians and other health care providers may already understand their patients in ways that clinical researchers do not, it could be useful to develop approaches to accessing information from these stakeholders without demanding more of their already limited time. Philip Alberti of the Association of American Medical Colleges noted that there are examples of community member involvement in protocol development, data analysis, and reporting that could help inform how the field moves forward with patient-engaged science. William Riley of the National Institutes of Health stated that the fields of behavioral intervention research and community intervention research might have applicable lessons.


Topic 2: Capturing Patient Perspectives and Preferences on Benefit–Risk

Identifying Gaps and Barriers

Some workshop participants discussed gaps and barriers in identifying and incorporating patient perspectives and preferences when it comes to benefit–risk of using medical products. Kevin Weinfurt of Duke University noted that there is a difference between choice and preference and it may be disingenuous to ask patients and their caregivers about preferences when their eventual choices may be limited given their social, cultural, and economic environment. Weinfurt said the process of eliciting and contextualizing patient preferences and choices could take place earlier in development, perhaps in the precompetitive space, rather than once product development is completed. He said a challenge to this approach is that many of the researchers most able to conduct such studies often do not realize the value in them or have difficulty obtaining the necessary funding. It may be beneficial to the field if patient preference research was incorporated into a common database that spans all diseases, rather than separating the information according to disease, said John Bridges of The Ohio State University College of Medicine.

Exploring Methods and Approaches

Individual workshop participants discussed methods and approaches that could be employed to address existing gaps and barriers in understanding patient preferences on benefit and risk. Some workshop participants considered approaches to make the collection of PPI easier and how additional resources for research could be supported. Weinfurt suggested that leveraging research networks to compile the large sample sizes needed for these types of studies could address this gap. He highlighted that workshop participants in his breakout session considered how pilot projects could illuminate study requirements and determine the value of characterizing context effects with greater precision. Riley suggested another way to elicit PPI could be to include relevant questions in large cohort studies. Tanisha Carino of FasterCures added that efforts to evaluate whether preferences matter could look at the market success of products that have evolved because of patient input, such as anti-retroviral therapy. Bridges mentioned that in his breakout session, workshop participants discussed how preference research could be used both to make population generalizations as well as to understand preference judgments at the individual patient–clinician level, such as in shared decision making.

Some workshop participants discussed tools to support research into patient preferences on benefit risk. Bridges and Dillon mentioned that a large preference study would require tools that could be used repeatedly, such as the PRO instruments developed for cancer adverse events,[18] and that ideally these preference measurement tools would be applicable to many diseases. Luther Clark of Merck suggested that research could be conducted on how tools from other industries could be adapted widely to evaluate patient preferences. He said Netflix and Google are examples of systems that offer services based on consumer preferences. Other workshop participants, including Liao, Skirboll, and Jessica Scott of GlaxoSmithKline, echoed the potential of artificial intelligence and machine learning as a tool for understanding—or predicting—patient preferences and decision making and encouraged partnerships with data scientists. To encourage the development of such broadly applicable tools, O’Callaghan said interested funders could consider adding a requirement to deliver a validated decision tool along with the results of a study.

Topic 3: Incorporating Patient Input in Clinical Trial Development and Continuous Improvement

Identifying Gaps and Barriers

Some workshop participants also considered gaps and barriers to incorporating patient input to clinical trial design. T. J. Sharpe of Starfish Harbor mentioned that workshop participants in his breakout session highlighted the lack of comprehensive data available on the patient experience in clinical trials. Sharpe emphasized the importance of understanding survey fatigue, survivor bias, and study dropout as they relate to trial participant experience.

Fowler suggested that understanding what motivates some clinical trial participants beyond their own treatment could be a means of helping other patients appreciate the global importance of participating in clinical trials.

Some workshop participants discussed tools and techniques that could better support incorporating patient input into clinical trial design. Sharpe stated that tools designed to measure the experience of trial participants—particularly as it affects recruitment, enrollment, attention, and trial cost and duration—would be beneficial. Dillon, as well as Marc Boutin of the National Health Council, suggested new statistical models may be needed to support better trial design and to segment data according to different patient profiles. Marni Hall of IQVIA said that workshop participants in her breakout session emphasized a need for validating the events of interest, data standards, and ontologies for systematic and structured data capture for post-market information in order to ensure the reliability of these tools in clinical trials.

Exploring Methods and Approaches

Individual workshop participants explored approaches that could potentially address gaps and barriers in incorporating patient input into clinical trial designs. Some workshop participants discussed approaches that trial sponsors could undertake. Bartek said that workshop participants in his breakout session considered how beginning discussions with trial sponsors as soon as they have a therapeutic candidate and before designing their protocols could be beneficial. Trial sponsors could use patient-generated natural history data to identify the optimal population for a particular therapeutic candidate and to define their inclusion and exclusion criteria, as well as primary and secondary endpoints, Bartek added. James Valentine of Hyman, Phelps & McNamara, P.C., suggested that patient preference questionnaires could help sponsors do a postmortem on the success of a trial. To encourage sponsors to include patient input when designing a trial, Bartek said the value added for patient engagement could be determined. Some workshop participants emphasized the benefits of standardization in these approaches. Sharpe stated that such standardized approaches for measuring patient experience could enable meaningful comparisons between trial protocols and produce actionable items for improvement. For example, he said, continuously monitoring a patient’s experience could be included with the continuous monitoring already specified for medical data. Fowler and Mullin noted that using standardized instruments and creating repositories for the data generated from those instruments could help advance the field.

Several workshop participants discussed specific areas in trial design where patient input could be incorporated and suggested approaches to capture and use that information. Stone suggested that prospective trial participants could be presented with vignettes about what a trial might be like as a means to estimate compliance and what potential barriers to completing the trial might be. Hall summarized a discussion from her breakout session on how trial participant informed changes to clinical trial design could improve the efficiency of data capture and potentially lower costs to addressing longer-term safety and effectiveness questions. Hall stated methods for soliciting trial participant input could include virtual trials, site-less design elements, and enriched studies that include PROs, EHRs, and claims data to power subpopulation analysis and cover a more generalizable population. Hall also emphasized the importance of trial participant consent and site selection processes to optimize efficiency and data governance regarding the use and reuse of data. Domitilla Masi of Avalere Health noted that because many of the important issues in trial participant experience are likely to be similar to issues that are important to patients when accessing any health care option, an area of research could be how patients’ experiences in clinical care could inform the choice of trial endpoints or other aspects of trial design. Finally, because there is a dearth of data on how medical devices perform in real-world populations, EHRs and claims data could be sources to help identify relevant outcomes, said Rachael Fleurence of the National Evaluation System for health Technology Coordinating Center.

Considering Stakeholders to Engage

Some workshop participants identified relevant stakeholders that could be engaged to facilitate incorporating patient input into clinical trials. Bartek stated that patient advocates could help ensure the procedures in a trial will be tolerable for patients participating in the trial. Patient scientists, who have an understanding of the development and regulatory environments as well as knowledge in a particular disease area, could work with the clinical team to ensure trials produce the necessary data (e.g., capture information needed to do a safety assessment) while also meeting trial participant needs and reducing unnecessary burden, said Cryer, Dillon, and Schroeder. Some workshop participants, including Bartek, Mullin, O’Callaghan, Sharpe, and Laura Lee Johnson of FDA, emphasized that regulatory agencies could act as a resource in trial design and help clarify regulatory requirements.


Carino observed that the workshop discussions highlighted the need for research to improve the state of the field and for the research to be open-source and precompetitive, though sharing existing best practices continues to be important. Advancing the science of patient input ultimately means facilitating change in training, skill sets, leadership, culture, and expectations. “Getting a research agenda is just a first step to begin to not only say what are the questions we should be answering, but who should be answering those questions,” said Carino.

Going forward, building a workforce that includes disciplines that may never have been involved in medical product development is an exciting opportunity for the field, said Carino. She said a key issue will be finding ways to understand the data about patient experiences, engage with partners to ensure that understanding is truly meaningful, and creating an expectation of how to triangulate the resulting knowledge with how medical product development has been done. Changing the culture to incorporate patient experience as part of medical product development, she added, will take years for most organizations.

[1] See (accessed October 18, 2018).

[2] See (accessed October 18, 2018).

[3] See (accessed October 18, 2018).

[4] See (accessed October 18, 2018).

[5] See (accessed October 18, 2018).

[6] See (accessed October 18, 2018).

[7] See (accessed October 18, 2018).

[8] See (accessed October 18, 2018).

[9] See (accessed October 29, 2018).

[10] See (accessed October 18, 2018).

[11] Ho, M. P., J. M. Gonzalez, H. P. Lerner, C. Y. Neuland, J. M. Whang, M. McMurry-Heath, A. B. Hauber, and T. Irony. 2015. Incorporating patient-preference evidence into regulatory decision making. Surgical Endoscopy 29(10):2984–2993.

[12] Rummel, M., T. M. Kim, F. Aversa, W. Brugger, E. Capochiani, C. Plenteda, F. Re, P. Trask, S. Osborne, R. Smith, and A. Grigg. 2017. Preference for subcutaneous or intravenous administration of rituximab among patients with untreated CD20+ diffuse large B-cell lymphoma or follicular lymphoma: Results from a prospective, randomized, open-label, crossover study (prefmab). Annals of Oncology 28(4):836–842.

[13] See (accessed November 7, 2018).

[14] See (accessed November 7, 2018).

[15] Lim, S. S., A. J. Kivitz, D. McKinnell, M. E. Pierson, and F. S. O’Brien. 2017. Simulating clinical trial visits yields patient insights into study design and recruitment. Patient Preference and Adherence 11:1295–1307.

[16] See (accessed October 18, 2018).

[17] See (accessed October 18, 2018).

[18] Basch, E., A. M. Deal, A. C. Dueck, H. I. Scher, M. G. Kris, C. Hudis, and D. Schrag. 2017. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 318(2):197–198.

DISCLAIMER: This Proceedings of a Workshop—in Brief was prepared by Joe Alper, Sylvia Ncha, and Amanda Wagner Gee, with assistance from Carolyn Shore, as a factual summary of what occurred at the workshop. The statements made are those of the rapporteurs or individual workshop participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.



Cynthia Grossman, FasterCures (Co-Chair); Marilyn Metcalf, GlaxoSmithKline (Co-Chair); Marc Boutin, National Health Council; Kenneth Getz, Tufts University School of Medicine; Mats Hansson, Uppsala University; Lynn Hudson, Critical Path Institute; Theresa Mullin, Food and Drug Administration; William Riley, National Institutes of Health; Roslyn Schneider, Pfizer Inc.; Suzanne Schrandt, Arthritis Foundation; Lana Skirboll, Sanofi; Pamela Tenaerts, Duke Clinical Research Institute; John Wagner, Takeda Pharmaceuticals; Richard Willke, International Society for Pharmacoeconomics and Outcomes Research.


*The National Academies of Sciences, Engineering, and Medicine’s planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published Proceedings of a Workshop—in Brief rests with the institution.


STAFF: Carolyn Shore, Director, Forum on Drug Discovery, Development, and Translation; Morgan Boname, Program Officer; Rebecca English, Program Officer; Amanda Wagner Gee, Program Officer; Sylvia Ncha, Associate Program Officer; Melvin Joppy, Senior Program Assistant.


REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Kathryn O’Callaghan, U.S. Food and Drug Administration, and T.J. Sharpe, Starfish Harbor. Lauren Shern, National Academies of Sciences, Engineering, and Medicine, served as the review coordinator.


SPONSORS: Abbvie Inc; American Diabetes Association; Amgen Inc.; Association of American Medical Colleges; AstraZeneca; Burroughs Wellcome Fund; Critical Path Institute; Eli Lilly and Company; FasterCures; U.S. Food and Drug Administration; Foundation for the National Institutes of Health; Friends of Cancer Research; GlaxoSmithKline; Johnson & Johnson; Merck & Co., Inc.; National Institutes of Health: National Cancer Institute, National Center for Advancing Translational Sciences, National Institute of Allergy and Infectious Diseases, National Institute of Mental Health, National Institute of Neurological Disorders and Stroke, and Office of the Director; New England Journal of Medicine; Pfizer Inc.; Sanofi; Takeda Pharmaceuticals.



For additional information regarding the workshop, visit


Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2018. Advancing the science of patient input in medical product research and development: Toward a research agenda: Proceedings of a workshop—in brief. Washington, DC: The National Academies Press. doi: