4

Interpretation and Application of Systematic Reviews and Meta-Analysis to Evaluate the Totality of Evidence

This chapter describes the presentations and discussions that took place during the third workshop, titled Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Systematic Reviews and Meta-Analysis to Evaluate the Totality of Evidence, which took place on October 3, 2023. The overall objectives of the third workshop were to

• Recognize the impact of risk of bias and publication bias on the interpretation of study results;
• Describe the impact of data errors on the conclusions of systematic reviews (SRs) and meta-analyses (MAs);
• Describe the process of evaluating the strength of the totality of evidence; and
• Describe the different applications of SRs and MAs to research and policy development and the unique considerations for each application of evidence.

The workshop featured two main speakers, Karima Benkhedda of Health Canada and Barbara O. Schneeman of the University of California, Davis. The final workshop was moderated by planning committee member Chizuru Nishida of the World Health Organization (WHO), retired, and included a panel discussion featuring the presenters and additional discussants, Elie A. Akl of the American University in Beirut and Vasanti Malik of the University of Toronto. This final workshop in the series concluded with closing remarks from planning committee chair Katherine L. Tucker of the University of Massachusetts Lowell. Workshop speakers addressed

the following questions, which were posed by the workshop sponsor in advance of the workshop:

• How to consider statistical heterogeneity when evaluating diet and disease relationships? Are higher levels of unexplained statistical heterogeneity acceptable for the field of nutrition? What are the best practices for addressing publication bias?
• How to consider risk of bias when evaluating diet and disease relationships?
• How can MA be used to evaluate the strength of the totality of evidence when there is evidence from different types of nutrition study designs?
• How can MA be used to evaluate the strength of evidence when different outcomes are reported in different studies?

FROM SCIENCE TO POLICY: EVALUATING NUTRITION EVIDENCE FOR INFORMED DECISION MAKING

Benkhedda’s presentation focused on evaluating nutrition evidence for informed decision making. Her presentation covered the use of SRs and MAs for the substantiation of health claims and policy and guideline development by Health Canada as case studies to illustrate the process. Benkhedda also addressed publication bias, heterogeneity, and risk of bias evaluation and their relevance for evaluating the totality of the evidence to inform policy and guideline development. Benkhedda disclosed that she is part of the NuQuest working group that develops risk of bias assessment tools for nutrition research, and her presentation was developed using both published research and unpublished data.

Benkhedda defined a “health claim” as any presentation in labeling or advertising that states, suggests, or otherwise implies that a relationship exists between the consumption of a food or ingredient in the food and a person’s health. She noted that such claims can be expressed in images or symbols and are not limited to words. Exploring Health Canada’s approach to evaluating evidence to support health claims, Benkhedda spoke about two major guidance documents that are available to inform industry about how to use the literature to inform their health claims: Health Canada’s Guidance Document for Preparing a Submission for Food Health Claims¹ and Health Canada’s Guidance Document for Preparing a Submission for Food Health Claims

___________________

¹ https://www.canada.ca/en/health-canada/services/food-nutrition/legislation-guidelines/guidance-documents/guidance-document-preparing-submission-food-health-claims-2009-1.html (accessed January 10, 2024).

Using an Existing Systematic Review.² Benkhedda said that while SRs are required to substantiate health claims, MAs are not. However, MAs can be useful for informing the review and evaluation of evidence.

Benkhedda detailed the systematic approach required for health claim substantiation,³ as shown in Figure 4-1. The process begins with a research question that is the basis of the claim, then follows a specific set of steps that culminates with a conclusion about the claim, development of the claim wording, and determination of the conditions under which the claim can be used.

Benkhedda explored the types of study designs that are used for the substantiation of health claims by Health Canada. She said that SRs, MAs, and randomized controlled trials (RCTs) are considered the highest level of evidence for health claims because they establish causality and provide information on intake-response relationships. She said that prospective observational studies (i.e., cohort studies and nested case-control studies) could also be included but would be considered a lower quality of evidence because they only show association, have more confounders, and cannot establish causality. She explained that prospective studies are more prone to bias, both through self-reporting and selection bias. On the topic of bias, Benkhedda added that publication bias may impact a review. Although this concern is not unique to MAs, she noted that it can be a major challenge due to the tendency to report on studies that show a significant effect. More bias can be introduced through search methods, such as language constraints or ignoring “gray” literature, which was previously described by Andrew Jones in Chapter 3. Benkhedda cautioned that having relevant studies missing from a review could adversely impact the intended decision making.

Benkhedda detailed a real-world example of an MA that was performed to examine the impact of dietary changes on low-density lipoprotein cholesterol. The MA included RCTs but contained high levels of statistical heterogeneity. In the forest plot, the studies were organized from largest to smallest effect to examine the relationship between sample size and effect size and the possibility of missing studies with small effect size or no effect, which could impact the results of the MA. To assess the absence of such studies, Benkhedda referenced the use of funnel plots, which were described by Jones in Chapter 3. Benkhedda noted that this method is imperfect and may perform poorly in settings with high heterogeneity because asymmetry in the funnel plot, which represents missing studies, can also occur because

___________________

² https://www.canada.ca/en/health-canada/services/food-nutrition/legislation-guidelines/guidance-documents/guidance-document-preparing-submission-food-health-claims-2011.html (accessed January 10, 2024).

³ Adapted from Health Canada, Guidance Document for Preparing a Submission for Food Health Claims, 2009.

of the high heterogeneity. Benkhedda said that it was important to use additional methods of analysis to assess the quality of evidence, especially since the review would be used to inform decision making in a policy setting. She said that the team in this example also used subgroup analyses to explore the heterogeneity present in their selected studies, examining the study duration, participant demographics, and other relevant subgroups. They also tried removing studies with the strongest effects to assess the impact of their absence on the overall results of the MA. The bottom line, Benkhedda said, is that interpreting publication bias involves a combination of visual inspection of funnel plots, statistical tests, sensitivity analysis, and expert judgment.

Benkhedda addressed a question of how to consider statistical heterogeneity when evaluating diet and disease relationships, inquiring whether higher levels of unexplained statistical heterogeneity are acceptable in the nutrition field. She stated that in situations where heterogeneity cannot be explained, a decision should be made about whether to pool the data. She said that some experts warn against pooling in high heterogeneity settings due to reduced confidence in effect estimates. Benkhedda acknowledged the validity of this concern in the nutrition field, even though high heterogeneity is often expected. She added that publication bias should also be considered, suggesting that research teams consult with both research methods and subject-matter experts. She said that it is important to evaluate each study individually rather than make a general statement about allowing for high levels of heterogeneity. Poorly done studies will reduce the overall quality of evidence and introduce bias, which will negatively impact the summary effect of the MA. Benkhedda shared Figure 4-2, which details the various domains of bias that may exist in nutrition studies (Kelly et al., 2022)

As Benkhedda discussed, most risk of bias tools are not specifically designed to address the key ways in which bias can enter nutrition studies, but some tools have been adapted to address these issues. To illustrate this, she displayed the two quality appraisal tools used by Health Canada⁴ shown in Figure 4-3. As Benkhedda highlighted, the quality appraisal tool for prospective observational studies, shown in Figure 4-3, includes questions such as whether the exposure was assessed more than once and whether the methodology used to measure the exposure was reported. She noted that confounders are also an area of concern in observational studies, and a question in the quality appraisal tool asks whether confounders were corrected for the study design or analysis process.

___________________

⁴ https://www.canada.ca/en/health-canada/services/food-nutrition/legislation-guidelines/guidance-documents/guidance-document-preparing-submission-food-health-claims-2009-1.html (accessed January 10, 2024).

Benkhedda described an additional example from Health Canada in which the evidence for a health claim on whole grains and coronary heart disease (CHD) was evaluated. The objective of the study was to determine whether the given evidence supported a health claim about whole grain foods and a reduced risk of CHD in the general population. The evaluation team used the Population, Intervention, Comparator, Outcome (PICO) framework to set up their SR, as shown in Box 4-1. The PICO framework was discussed in more detail by Celeste Naude in Chapter 3.

A systematic review of 26 RCTs and 6 cohort studies was conducted, and an MA of RCTs was performed for LDL and total cholesterol outcomes (Health Canada, 2012). Evidence from the RCTs showed a statistically significant effect, but when analyzing only the high-quality evidence, or the evidence on grains other than those high in beta-glucan fiber, this effect

was lost. The data from cohort studies was too heterogeneous to be pooled, and most of the studies were deemed to be too low quality with a high risk of bias due to lack of adjustment for confounders and lack of control for potential confounders. In the end, the evidence was not sufficient to support a health claim about whole grains and CHD risk reduction.

Benkhedda addressed the question of how to consider risk of bias when evaluating diet and disease relationships. She stated the importance of considering the overall quality of evidence and referenced the previous example of the MA on whole grain intake and CHD in which the evidence was not considered high enough quality to substantiate a claim. Referring to Figure 4-1, which depicts Health Canada’s approach to health claims substantiation, Benkhedda spoke about ways to assess causality, including considering the overall consistency of the evidence, the strength of association, and the intake-response (or dose-response) relationships. She explained that MAs can help to determine health claim validity by assessing these factors and the overall quality of evidence across studies. MAs can answer questions such as whether consistency was high across the higher quality studies and if appropriate tests were used to quantify heterogeneity. Benkhedda suggested analyzing the proportion of studies that showed statistically significant effects and explore their quality and what factors might have impacted the statistical significance of the nonsignificant studies. Benkhedda said that when considering dose-response relationships, the minimum effective amount shown to produce a response should be identified, and in observational studies, statistically significant differences found between the highest intake groups and lowest intake groups should be examined.

Benkhedda spoke about the importance of assessing the generalizability and relevance of study data to the general population when looking

to inform guideline and policy development. One question to consider is whether the population studied is representative of the general population. Benkhedda emphasized the importance of establishing the applicability of the MA results to the target population, and for health claims, the target population is the general population of the country. Doing subgroup analyses within the MA can help to establish the robustness of the effect for important subgroups of interest with a focus on the physiological meaningfulness of a food’s effect. Questions to consider include whether the observed effects have clinical relevance and whether the effect disappears within a few weeks or months or is durable long-term. Benkhedda highlighted the importance of quantifying the impact of the food exposure on human health.

Addressing a question of how MAs can be used to evaluate the strength of the totality of evidence when the evidence comes from different types of nutrition study designs, Benkhedda reiterated the benefits of a broad, holistic assessment of evidence. Consider the comprehensiveness, relevant outcomes, consistency and strength of association, quality, meaningfulness of the effect size, and the generalizability of the data, she suggested.

Benkhedda also addressed the question of how MAs can be used to evaluate the strength of the evidence when different outcomes are reported in different studies. To this, Benkhedda said that claim wording should reflect the specific evidence. For example, the claim should reflect whether the evidence showed prevention of disease, change in disease outcome, or change in disease risk biomarkers. She illustrated this point with another real-world example. Health Canada examined the evidence to support a health claim for fruit and vegetable consumption to reduce the risk of heart disease and examined evidence to support a health claim of whole grains consumption to reduce CHD risk. They found sufficient evidence to support the health claims for fruits and vegetables but did not find sufficient evidence to support the health claims for whole grains. The lack of sufficient evidence was due to the fact that a sensitivity analysis revealed the evidence for the whole grains health claim was in limited trials on grains high in beta-glucan and in studies judged to be of poor quality, which were credited with producing most of the effect.

Benkhedda concluded her presentation with a general overview of the considerations for use of MA in policy development. She suggested using the best evidence available, including SRs, MAs, and other relevant individual studies. Considerations should be given to the relevance of a study to the specific policy question, the overall quality of evidence, the level of certainty, and the applicability of the evidence to the general population in a national context. The advantage of MAs and SRs, she noted, is that they examine a large sample of studies under similar conditions and can draw conclusions that are relevant to policy development. However, Benkhedda

noted that the included research should be relevant to the policy question, and she suggested that it may be helpful to conduct SRs in collaboration with policy makers.

NUTRITION AND POLICY: EVALUATING THE EVIDENCE

Barbara O. Schneeman’s presentation focused on examples of the application of MAs and SRs in nutrition policy. She featured examples across three bodies that use nutrition research to develop guidelines and policies: the U.S. Food and Drug Administration (FDA), the U.S. Dietary Guidelines Advisory Committee (DGAC), and the WHO Nutrition Guidance Expert Advisory Group Subcommittee (NUGAG). Specifically, she addressed FDA’s use of SR evidence in acceptance, denial, or qualification of health claims; DGAC’s use of SR evidence to produce DGAC reports to advise the relevant federal agencies on updates to the Dietary Guidelines for Americans (DGA); and WHO’s use of SRs and MAs in the development of nutrition guidelines. Schneeman disclosed that she holds, or has held, affiliations with numerous stakeholder organizations, including DGAC, WHO NUGAG, National Academies of Sciences, Engineering, and Medicine’s Food and Nutrition Board, and the International Union of Food Science and Technology Task Force on Food Classification. She also noted her position as an advisory board member, and/or member of the board of trustees for organizations including the International Food Information Council, McCormick Science Institute, and the Coalition for Grain Fiber Science Advisory Committee.

Schneeman provided WHO’s definitions of SR and MA. As she quoted, “an SR is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research and to extract and analyze data from the studies that are included in a review” (WHO, 2014). An SR is different from an MA, which refers to the quantitative synthesis, or pooling, of outcome data across comparable studies to achieve a pooled estimate of effect (WHO, 2014). Schneeman explained that if data from an SR meets certain requirements, such as high homogeneity across study design, population, and intervention, then the data can be combined into an MA. An MA can also be distinguished from an SR in that an MA is a statistical method that provides a summary estimate of effect across a body of evidence.

Schneeman described the process of undertaking an SR by each of the three authoritative bodies. At FDA, this process is initiated in response to a petition or when the agency considers updating existing claims. A scoping review is useful to determine whether there are any major and relevant omissions in the literature submitted with a petition. The DGAC uses SRs to review the evidence on topics for inclusion in the next iteration of the DGA,

and the U.S. Department of Agriculture’s (USDA’s) Nutrition Evidence Systematic Review (NESR) team⁵ has proposed using scoping reviews to identify potential topics for the DGA. At WHO, an SR is used to identify the availability of relevant evidence for guideline development and to facilitate protocol development.

Schneeman detailed the process of structuring an SR and an MA. She noted that her description would not be comprehensive but instead illustrative for the topic of interest. She stated that research teams should identify the population of interest, including characteristics such as age, sex, and health status, and the intervention or exposure, aiming for specificity to allow for identification of studies that are relevant to the policy question. She noted that for health claims, FDA would be particularly interested in the impact of a substance compared to the lack of that substance. Likewise, for dietary guidelines the nature of the comparison is important. For example, when considering the impact of saturated fat intake on health outcomes, reviewers must consider the comparison group, such as low fat, other types of fat, or carbohydrates. It is a challenge, Schneeman noted, to be clear about the exact nature of the intervention, the control, and the comparison.

Schneeman explained the importance of identifying the outcomes of interest for the review and how to decide what evidence to include and exclude from the review. As mentioned by Naude and Hooper in Chapter 2, having protocols in place can be beneficial for ensuring that all decision-making criteria are consistently applied. Specific, meaningful outcomes are important when evaluating evidence for health claims, Schneeman said. She emphasized that it is critical to consider the intended use of evidence and which studies are and are not relevant to the research question. For example, when developing dietary guidelines for the national population, a study that showed the clinical impact of a dietary intervention used to treat a disease may be relevant to evaluate clinical treatments but not relevant for developing policies to reduce risk for a disease.

Schneeman noted that it is important for policy makers to understand the overall quality of evidence and whether it results in significant scientific agreement. To this end, Schneeman described how an MA can provide insight into the inconsistencies in the available evidence. As described by Wells in Chapter 3, Schneeman noted that MAs can be used to examine the differences between studies. She restated the usefulness of subgroup analyses, previously explained by Jones and Wells in Chapter 3, when subgroups are relevant to the specific guidelines being developed. She noted that DGAC and the USDA NESR team provide an analytical framework for

___________________

⁵ For information on NESR, see https://nesr.usda.gov/ (accessed January 10, 2024).

understanding the key factors that may impact the relationships in nutrition studies, such as confounders, covariates, and moderators.

Schneeman described the processes of structuring an SR at the three entities of focus. At FDA, when an SR is developed in response to a health claim petition or as evidence for food labeling, the petition will be used to create the PICO elements, and the inclusion and exclusion criteria are specified in the FDA guidance document.⁶ For DGAC, NESR methodology specifies the inclusion and exclusion criteria that will be used in the SR, which must be relevant to the DGA and may be modified as needed by DGAC to address specific questions. The SR is conducted by methodological experts. Figure 4-4 displays the steps used by DGAC to implement NESR’s process⁷ for planning and executing an SR. When an SR is performed for WHO NUGAG, the subcommittee determines the PICO elements, identifies the outcomes that are critical for decision making, and specifies the inclusion and exclusion criteria using an approach that is consistent with the WHO Guideline Development handbook (WHO, 2014). Overall, Schneeman reinforced the importance of bringing methodology and subject-matter experts into every level of the process.

Schneeman discussed the methods for determining the strength of the evidence and how these processes differ across the three governing bodies. At FDA, the focus is on whether the evidence is consistent with significant scientific agreement. DGAC assigns each SR a grade with criteria for risk of bias, consistency, precision, directness, and generalizability. WHO uses the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology for rating evidence, which considers study design and allows for researchers to evaluate and grade the quality of their evidence on a rating scale from very high-quality evidence to very low-quality evidence. This approach, Schneeman said, may help to improve objectivity when assessing the strength of the evidence.

After grading and analyzing the evidence, Schneeman explained, the next phase is the decision-making process, a step that she described as critical but often overlooked. She explained that this phase of the process is where the three groups most differ in their approach. At FDA, the criteria and conditions related to the food products that might bear a claim must be considered as well as the legal and economic factors on the use of claims in labeling. DGAC considers not only the evidence from SRs for its conclusions but also data analysis from National Health and Nutrition Examination Survey to understand current intakes as well as food pattern modeling to structure healthful dietary patterns. Schneeman explained that

___________________

⁶ For the FDA guidance document, see https://www.fda.gov/food/food-labeling-nutrition/qualified-health-claims (accessed January 10, 2024).

⁷ https://nesr.usda.gov/ (accessed January 10, 2024).

DGAC integrates the evidence from these three approaches to develop its conclusions. For example, the evidence from SRs to support the DGA recommendation on added sugars was graded as limited and insufficient. However, examining food pattern modeling allowed DGAC to estimate the extent to which added sugars could be consumed while maintaining a healthful dietary pattern. This additional information allowed DGAC to conclude that typical dietary intakes could be improved by recommending a limit to added sugars intake. DGAC was able to come to this conclusion by integrating the full body of evidence. At WHO, SRs and MAs are integrated into decision making using a series of contextual factors to determine the strength of the recommendations. For example, WHO considers the balance of benefits and harms, resource implications, confidence in effect estimates, equity and human rights, accessibility, and feasibility.

Schneeman shared some observations from her experiences using MA in the decision-making process. She stated that a well-designed SR is essential for a proper MA. She recommended having experts in methodology involved in designing the SR and performing the MA. She also recommended conducting a scoping review prior to an SR to set the parameters for the SR and focus on the information that is critical to the decision making. Schneeman said that SRs should be designed to fit the specific purpose of the policy or guidance, and MAs should allow for a deeper examination of the evidence to better understand the strength of the evidence. Again, she noted that subgroup analyses can assist with addressing and understanding high levels of heterogeneity. She also noted that MAs can provide a more objective assessment of the strength of the evidence, making them a useful tool. However, Schneeman said, when MAs are not possible, other tools can be used such as harvest plots. Harvest plots are often used when creating nutrition policy and guidelines in conditions where MAs are not an appropriate tool.

Schneeman provided a brief overview of the opportunities and challenges for using MAs to develop guidance across the three governing bodies profiled. She suggested that, at FDA, a graphic display (e.g., a forest plot) might better illustrate the balance of evidence and be a useful tool to clarify inconsistencies in the current evidence. For DGAC, MAs could provide more transparency when rating the strength of the evidence. She also suggested that nonqualitative summary tools could facilitate recommendations related to the food environment and policy and that subgroup analysis could better help the committee understand which subgroups would benefit most from specific changes to the DGA. Schneeman noted that MAs are used for guideline development at WHO, which allows for experts in methodology to assess the strength of the evidence and for expert guidance committees to use the MAs in the development of the recommendation.

She pointed out that MAs could also be used to determine when evidence is insufficient to produce a recommendation.

PANEL DISCUSSION

Nishida led a panel discussion featuring Benkhedda and Schneeman, who were joined by Malik and Akl.

Tools to Assess the Quality of Nutrition Research to Inform Policy

A question posed by planning committee member Russell Jude de Souza, and echoed by other audience members, asked whether the field of nutrition is so unique that it requires its own set of statistical and analytical tools. Nishida further inquired whether there are formal criteria for use of existing tools that would make them more effective in nutrition research for policy development. Akl replied, disclosing that his role as part of the GRADE working group has informed his perspective, and said that while many fields may consider theirs to be unique, his opinion is that the principles should be the same across fields. Whether a research team uses GRADE or Nutri-Grade to assess their evidence, the result should be the same. The general principles across fields are that certainty in effect estimates is affected by risk of bias, inconsistency of results, imprecision, indirect evidence, and publication bias. However, Akl said, in certain fields small effect sizes may be more meaningful. For example, in public health, a small effect size across a large population could lead to a meaningful public health impact. What is judged as a small effect size in one field might be considered significant in another field.

Akl described additional factors that he thinks are particularly relevant to the nutrition field. For example, in nutrition it is common to have outcomes with continuous measurements. Continuous scales often show heterogeneity due to the nature of the data. This factor should be considered when assessing heterogeneity in nutrition research, Akl said, and not “over-rated,” which could lead to the downgrading of the certainty of evidence. He referred to Schneeman’s comment about fitting the method to the purpose, further emphasizing that “the method should fit the purpose of the eventual product.” He noted that FDA and DGAC use different approaches for different goals: validation of health claims and development of guidelines, respectively. Akl highlighted that the involvement of content experts in the development and use of SRs and MAs is important, a point on which Malik concurred. Akl noted that experts are important for judging effect sizes and non-health effects.

Akl also encouraged the consideration of the contextual factors that impact nutrition policy and guideline development, such as the resources that would be required to implement or enforce a policy or recommendation

and the acceptance of a recommendation to stakeholders such as industry and consumers. He urged researchers to consider the real-world implications of a potential policy or guideline, suggesting that SRs that examine these contextual factors could be part of the methodologies that governing bodies use to inform their process. Akl noted that Health Canada’s process of analyzing research to inform policy development, as described by Benkhedda, is an excellent example of the consistent and effective use of existing research tools.

Malik suggested that the “hierarchy of evidence” pyramid, as shown in Figure 4-5, which was presented by Benkhedda and is commonly used when evaluating quality of evidence, may not be the ideal reference point for the field of nutrition (Yetley et al., 2017). Nishida concurred with this comment. Malik suggested that, given that the field of nutrition often examines longitudinal relationships between diet and health, a restructuring of the pyramid should be considered. Currently, RCTs are the highest level of evidence and cohort studies are viewed as lower quality; Malik suggested that they could be shown side by side. When it comes to nutrition, longitudinal cohort studies are often better able to truly examine longitudinal relationships, and it is typically not feasible to use RCTs for this purpose. Blinding is also not possible in most nutrition studies.

Benkhedda agreed that sometimes well-designed and well-conducted observational studies do yield better results than poor-quality RCTs. However, she noted that in validating health claims, Health Canada requires RCTs first and then observational studies can provide additional support. If an observational study shows an effect, a follow-up RCT on the food or nutrient in question can help to clarify and determine whether a causal relationship truly exists. However, she agreed that for dietary guideline development, RCTs may not be the most important source of data, suggesting that further discussion on this topic among nutrition experts may be warranted.

Schneeman stated that the GRADE approach can be helpful because it allows for the upgrading or downgrading of certain studies relative to their efficacy in answering the research question. According to Schneeman, the nutrition field does not need new or unique tools but can use existing tools to rate evidence as accurately as possible. She also emphasized that different government bodies, and different countries, have specific legal and contextual factors to consider when developing nutrition policy. For example, in the United States, the First Amendment is a consideration in the health claim process. RCTs may still be required to verify health claims. Schneeman added that while she used to think nutrition studies were unique in their complexity and required unique tools, her thinking has evolved over her career, and she has come to understand that using well-defined tools consistently over time to facilitate improved understanding of the data can contribute to effective policy decision making as well as to transparency in the decision-making process.

Benkhedda and Malik added their agreement with Schneeman and Akl that the consistent use of existing tools is key to their efficacy. Benkhedda also noted that many recent publications have initiated discussions to address consistency and quality of data and appropriate data analysis tools in the field of nutrition research. For example, one common problem is the lack of appropriate risk of bias tools that work with the common study types used in nutrition research. These tools can be adapted but may not work consistently across SRs and MAs, which limits how researchers can interpret bias in nutrition studies.

Conflict of Interest and Bias

The second major topic of conversation was the potential impacts of conflict of interest, including funding sources and other vested interests, on nutrition research results and ways to mitigate this bias. Benkhedda explained Health Canada’s approach for health claims validation, stating that Health Canada examines evidence through a set of objective criteria, and that while they do not exclude evidence due to funding source, they evaluate the merit of the study based on the same objective criteria used to evaluate all studies. Health Canada also considers whether the funding source may have influenced the outcome of the study. For nutrition guidelines, Benkhedda noted that the committee developing the guideline may choose to exclude studies funded by industry.

Akl added that funding by industry or an interested party, or a conflict of interest from study authors, does not necessarily negatively impact a study; but it has to predict when it would and when it would not. Therefore, these represent a “red flag” that should be attended to because evidence shows that studies funded by industry have more favorable results than those that are not. He also explained that conflicts of interest can exist beyond funding, such as intellectual conflicts of interest among members of a committee or a reviewing body. If an expert comes to a panel with a preconceived notion about a recommendation, Akl explained, they may not have an open mind about the evidence being presented. This type of conflict of interest can be very challenging to detect, as it falls outside the standard conflict of interest declaration and management policies.

Schneeman added that for health claims and petitions to FDA, most of the evidence provided will be funded by industry. However, the evidence must still meet prespecified transparent criteria, which are created to ensure that the evidence is high quality.

Malik added that she has found it useful in her experience as a researcher managing MAs to conduct a subgroup analysis by funding source. Also, the Nutri-Grade system has the option to screen data by funding source when using the tool to rank evidence in an MA.

This page intentionally left blank.