1

Introduction¹

In September and October 2023, the National Academies of Sciences, Engineering, and Medicine’s (the National Academies’) Food and Nutrition Board held a three-part virtual workshop series to discuss best practices for conducting meta-analyses (MAs) in nutrition research and utilizing MAs to inform policy. The three-part series, titled Use of Meta-Analyses in Nutrition Research and Policy: A Workshop Series, explored the evidence on methods for conducting, interpreting, and integrating the results of MAs for use in nutrition research, developing nutrition policy, and informing nutrition regulatory decision making.

The workshop series was sponsored by the U.S. Food and Drug Administration (FDA) and featured invited presentations and discussions with researchers, government officials, and other global leaders in nutrition research and policy. The workshop Statement of Task can be found in Box 1-1. The agendas for each workshop are presented in Appendix A, and Appendix B contains the biographical sketches of the speakers and moderators.

The workshop series included three 2-hour workshops that took place over a 3-week period and was planned by a committee, which was led by Katherine L. Tucker of the University of Massachusetts Lowell and

___________________

¹ The planning committee’s role was limited to planning the workshop, and the Proceedings of a Workshop was prepared by the workshop rapporteurs as a factual summary of what occurred at the workshop. Statements, recommendations, and opinions expressed are those of individual presenters and participants, and are not necessarily endorsed or verified by the National Academies of Sciences, Engineering, and Medicine, and they should not be construed as reflecting any group consensus.

comprised of leading experts in nutrition research and policy. Prior to the workshop series, FDA staff provided the planning committee with a list of questions to be addressed throughout the workshops. The questions for the first workshop are:

• What are best practices in identifying and avoiding extraction errors and errors in calculating mean differences and confidence intervals from the primary studies that are included in an MA, given that these errors are common in published literature?
• How should an MA be evaluated for methodological quality when extraction and/or data errors are present? At what point do data

• errors (in kind and number) reach a level that invalidates the conclusions of the MA?
• How to consider risk of bias when evaluating diet and disease relationships?
• What are the best practices for addressing publication bias?
• What criteria should be used to determine whether individual nutrition studies have too many clinical or methodological differences (e.g., treatment, dose, population, mean body mass index, duration, comparators/diets, results) to be combined into the same MA?
• What are best practices for planning appropriate subgroups and sensitivity analysis a priori?
• How can MA be used to evaluate the strength of the evidence when different outcomes are reported in different studies? And how can MA be used to evaluate the strength of the totality of evidence?

The questions for the second workshop are:

• Extraction errors from the primary studies that are included in MAs are common. What are best practices for avoiding/identifying these types of errors?
• What are the best practices for addressing publication bias?

The questions for the third workshop are:

• How to consider statistical heterogeneity when evaluating diet and disease relationships? Are higher levels of unexplained statistical heterogeneity acceptable for the field of nutrition? What are the best practices for addressing publication bias?
• How to consider risk of bias when evaluating diet and disease relationships?
• How can MA be used to evaluate the strength of the totality of evidence when there is evidence from different types of nutrition study designs?
• How can MA be used to evaluate the strength of evidence when different outcomes are reported in different studies?

As previously stated, the goals of the workshop series were to explore issues and best practices around MA. However, because MAs and systematic reviews (SRs) are closely related processes, they share many of the same important considerations. In light of this, many presentations included broader discussions around issues related to both MAs and SRs.

OVERVIEW OF THE WORKSHOP SERIES

The first workshop in the series, held on September 19, 2023, focused on the planning and execution of MAs for nutrition research. Workshop objectives included clarifying criteria for the selection of studies to be included in an MA, with a focus on the Population, Intervention, Comparator, and Outcome (PICO) framework, and planning for subgroup analyses. The agenda for the first workshop included opening remarks from planning committee chair Katherine L. Tucker, background information on the topic and the goals for the workshop, presentations from invited speakers Crystal Rivers and Sarah Gebauer from FDA, Celeste Naude of Stellenbosch University, and Lee Hooper of the University of East Anglia, and a panel discussion featuring Hooper, Naude, and additional discussants. Naude guided participants through the planning of SRs and MAs and addressed topics such as how SRs and MAs can be used to evaluate the strength of the evidence when different outcomes are reported in different studies. In the second presentation, Hooper focused on the pillars of planning that can help reduce the risk of bias in an SR or MA. Through this lens, Hooper addressed topics of how to consider risk of bias when evaluating diet and disease relationships and the best practices for addressing publication bias. The panel discussion featured additional discussants, including Sydne Newberry of the RAND Corporation and Christopher Schmid of the Brown University School of Public Health. The discussion was moderated by planning committee member Amanda J. MacFarlane of Texas A&M University. The closing remarks for this first workshop were delivered by planning committee member Mei Chung of Tufts University.

The second workshop featured presentations on best practices in conducting SRs and MAs, especially in the context of nutrition research and policy development. The objectives for the session were to explain the best practices for screening data for potential errors, how to use a system of evaluation for understanding risk of bias in study design, and how to systematically evaluate study results, including understanding the precision of estimates and the potential for publication bias. Finally, the workshop explored how to effectively interpret the results of an MA using a variety of statistical methods, what to do if assumptions are violated, and how to address the issue of high statistical heterogeneity, which is common in nutrition studies. The agenda for the 2-hour workshop included presentations from Emma Boyland of the University of Liverpool, Andrew Jones of Liverpool John Moores University, and George A. Wells of the University of Ottawa. Following the presentations, a panel discussion featured the presenters and additional discussants. The discussion included questions and contributions from Elie A. Akl from the American University of Beirut, Joseph Beyene of McMaster University, and M. Hassan Murad of the Mayo

Clinic. The discussion was moderated by planning committee member Janet A. Tooze of the Wake Forest University School of Medicine, and closing remarks were given by planning committee member Russell Jude de Souza of McMaster University.

The third and final workshop in the series featured presentations on best practices in interpretation and application of SRs and MAs, especially in the context of nutrition research. The objectives of the session were to recognize the impact of risk of bias and publication bias on the interpretation of study results, to describe the impact of data errors on the conclusions of SRs and MAs, to describe the process of evaluating the strength of the totality of evidence, to describe the different applications of SRs and MAs to research and policy, and to consider the evidence evaluation for each. The 2-hour workshop included presentations from Karima Benkhedda of Health Canada and Barbara O. Schneeman of the University of California, Davis. Following the presentations, a panel discussion, moderated by planning committee member Chizuru Nishida of the World Health Organization, retired, featured remarks from the presenters as well as Vasanti Malik from the University of Toronto and Elie A. Akl from the American University of Beirut. Closing remarks were given by Katherine L. Tucker. She also presented the final remarks for the entire workshop series.

Although each workshop had a distinct focus, several topics were reiterated throughout the series. For example, Hooper discussed ways to examine and reduce risk of bias, and Boyland explored methods of anticipating and avoiding bias in study design. Jones and Wells discussed the many ways in which statistical analyses can fail to address the unique needs of nutrition research, and Schneemann emphasized the benefits of consistent use of statistical tools to improve statistical analysis in the complex field of nutrition research. Discussion also focused on whether the field of nutrition research requires unique tools for evidence synthesis and decision making. In the closing remarks of the workshop series, Tucker noted that the complexities of nutrition research may require distinctive considerations and suggested that the best practices described throughout the series could inform future nutrition research and policy development. She stated that the field of nutrition research is not only uniquely complex but also uniquely situated to have a real-life impact on large populations through evidence-based policy development.

OPENING REMARKS FROM WORKSHOP SPONSOR

At the beginning of the first workshop, two speakers from FDA explained why their agency sponsored the workshop series. One example they gave is that MAs are increasingly reported in the scientific literature in

the nutrition field, and stakeholders are increasingly requesting that federal agencies consider MAs in support of nutrition policy and regulatory decisions. Crystal Rivers and Sarah Gebauer each spoke about FDA’s investment in the workshop series and the unique research and policy development challenges faced by the field of nutrition.

Rivers spoke about the use of MAs in the development of nutrition regulatory decisions, nutrition policy, and dietary recommendations and how MAs can be used to identify existing gaps in nutrition research. She noted that the use of MAs with SRs is becoming more common and that there has been a lack of discussion around how to best use and conduct MAs in the field of nutrition. Rivers explained that it was the hope of FDA that this workshop series would help inform a set of best practices for the use of MAs in nutrition policy development moving forward.

Gebauer emphasized the unique needs of FDA when it comes to their nutrition regulatory framework and how evidence is used to substantiate health claims by showing a direct relationship between a food product and a health outcome or disease risk. She gave an example of the nutrients calcium and vitamin D and their relationship to osteoporosis. Gebauer noted that when evidence for a health claim is weak, the strength of the evidence must be reflected in the language of the claim. She also pointed out some of FDA’s distinctive considerations when conducting their own reviews of the evidence related to nutrition labeling and health claims. For example, FDA typically conducts health claim reviews in response to stakeholder petitions, which identify the topic of the review based on stakeholder interests; and FDA relies on publicly available data for their health claim reviews. Gebauer highlighted the need for consistent application of tools and a set of best practices for conducting MAs and for evaluating and interpreting MAs in nutrition research and policy. She noted that FDA hoped to use the discussions from the workshop series to inform the development of clear recommendations for evaluating the quality of existing MAs, conducting MAs, and interpretating and integrating outcomes of MAs to evaluate the strength of a body of evidence for the development of nutrition policy.