2

Methods and Presentation

Starting in 2011, the National Toxicology Program (NTP) Office of Health Assessment and Translation (OHAT) began developing and incorporating systematic review methods into literature evaluations to assess scientific evidence of human health effects of exposures to environmental chemicals, physical substances, or mixtures (Birnbaum et al. 2013; Rooney et al. 2014). That effort was part of a cultural change within the environmental-health field in which approaches to evaluation of scientific evidence resulted predominantly in expert-based narrative reviews. However, mounting empirical evidence that narrative reviews generally lacked the ability to evaluate evidence in a rigorous, systematic, transparent, and reproducible manner indicated that more rigorous approaches to evidence-based decision-making were needed (Reenie and Chalmers 2009; NRC 2011; Woodruff and Sutton 2011). As a result, several agencies and institutions have undertaken the development and implementation of systematic review methods to address environmental questions (EFSA 2010; Woodruff and Sutton 2011, 2014; Murray and Thayer 2014).

NTP OHAT systematic review methods are described in several documents. First, the OHAT handbook on systematic review (published in 2015 and updated in 2019) represents the “standard operating procedures” for how systematic review and evidence integration are to be conducted for OHAT literature-based assessments (NTP 2015, 2019a,b).¹ Those operational guidelines are based largely on empirically tested approaches and expert input from various fields, such as the clinical sciences, including Cochrane (Higgins et al. 2019), Grading of Recommendations Assessment, Development, and Evaluation (Guyatt et al. 2008), the Navigation Guide (Woodruff and Sutton 2011, 2014), the Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental

___________________

¹ Although the protocol for the fluoride monograph refers to both versions of the OHAT handbook, the committee assumes that it was based on the updated version given that it incorporates a revised figure (NTP 2017, Figure 3, p. 20) that was provided in the updated version.

Studies (CAMARADES),² and the Systematic Review Centre for Laboratory Animal Experimentation (SYRCLE).³

For NTP OHAT literature-based assessments, such as the monograph evaluated in the present report, a protocol is developed and shared publicly before beginning the assessment. A protocol is the detailed plan that is to be followed in a specific systematic review and describes the rationale, the objectives of the review, and the conduct of each step of the review (NTP 2015; Higgins et al. 2019). The steps include problem formulation, which results in the development of the Population, Exposure, Comparator, and Outcomes (PECO) statement; development of inclusion and exclusion criteria for study selection; a search of the literature; extraction of data from included studies; critical appraisal of studies for risk of bias; synthesis of results from included studies; and hazard identification by integration of the evidence from human and animal studies and consideration of supporting evidence from mechanistic studies. Ideally, the protocol should follow guidance provided by the OHAT handbook but include details specific to the given systematic review, such as how to rate risk of bias in assessing outcomes of neurodevelopmental and cognitive effects as in the case here.

The methods section in the monograph should also describe how a specific systematic review was conducted. Ideally, the methods described in the monograph should align with the details outlined in the protocol or should transparently and explicitly document, describe, and justify any deviations from the protocol. In the present chapter, the committee provides its assessment of the methods and overall presentation.

GENERAL ASSESSMENT

The committee noted several strengths of the monograph. As noted, the protocol is a critical component of a systematic review and ideally minimizes reviewer bias, allows feedback at early stages of the systematic review, and transparently highlights any changes made as the systematic review process unfolds (IOM 2011). The protocol for the monograph contains descriptions of each step of the systematic review and clearly outlines several protocol revisions, including the date and justification of each change. Furthermore, changes in the protocol are clearly indicated so that both the original text and the modified text are readily

___________________

² CAMARADES provides a supporting framework for groups involved in the systematic review and meta-analysis of data from experimental animal studies. See http://www.dcn.ed.ac.uk/camarades/.

³ SYRCLE focuses on the execution of systematic reviews of animal studies aimed at more evidence-based translational medicine. See https://www.radboudumc.nl/en/research/departments/health-evidence/systematic-review-center-for-laboratory-animal-experimentation.

apparent. NTP appears to be adhering to best practices for systematic reviews with respect to the availability and documentation of such a protocol before initiation of a review.

The committee also was impressed by the availability of systematic review data in the interactive, freely available program Health Assessment Workspace Collaborative (HAWC).⁴ Given the plethora of data extracted from studies and the risk-of-bias ratings and justifications, interactive programs increase the ease with which an independent reviewer can explore the data in more detail without being limited to the graphics and tables provided in the report itself. The committee appreciated that most of the tables and figures in the monograph were available with additional study details and interactive graphics in HAWC.

The committee, however, had some overarching concerns regarding the protocol, data presentation, and communication that are described in the following sections, and it provides some suggestions for improvements.

PROTOCOL

The protocol for the systematic review described in the monograph was published on NTP’s Web site in June 2017 and made available for public comment. It was reviewed several times by technical advisers selected for their expertise on this topic. In general, it describes the overall systematic-review process and clearly outlines modifications that were made during the review. The committee, however, identified several issues associated with the protocol. First, the role of the OHAT handbook in developing the protocol is unclear. Second, important details are missing from the protocol. Third, inconsistencies between the protocol and the monograph raise concerns. Those issues are discussed further below; other issues associated with execution of the protocol are discussed in Chapters 3 and 4. Given the issues raised here and in later chapters, the committee finds that there are some deficiencies in the protocol and its execution.

Role of the OHAT Handbook

As discussed, the OHAT handbook outlines “standard operating procedures for systematic review and evidence integration for conducting OHAT literature-based assessments” (NTP 2019a, p. v) and is intended as a “living document” that is continually updated to reflect refinement and modifications of the OHAT approach. However, the protocol scarcely refers to the OHAT handbook and does not discuss its role in developing the protocol. Specifically, the only references to the OHAT handbook in the protocol are noted below.

• A statement that “the systematic review will be based on guidance outlined in the Office of Health Assessment and Translation (OHAT) Handbook for Conducting a Literature-Based Assessment” (NTP 2017, p. 3).

___________________

⁴ See https://hawcproject.org/user/login/?next=/portal/.

• Supporting descriptions of “tier 3” studies that have overall very serious risk-of-bias concerns for individual epidemiology studies (NTP 2017, pp. 8, 10).
• Further guidance for assessing confidence in the body of overall evidence (NTP 2017, p. 13).

Thus, the role of the OHAT handbook in developing the protocol is unclear. That ambiguity leads to concerns about the lack of detail in the protocol and about apparent conflicts between the methodologic approach in the protocol and the OHAT handbook itself. Several examples of that ambiguity and the associated concerns are provided below.

• Nomination history. According to the OHAT handbook, this section should describe “the history of the nomination…steps the NTP has taken to solicit feedback on the topic under consideration, including Federal Register notices, requests for information in the NIH Guide for Grants and Contracts, outreach to federal agencies on the NTP Executive Committee, or outreach to other divisions within NIEHS [National Institute of Environmental Health Sciences]” (NTP 2019a, p. 12). This section should also provide a summary of any comments received during the comment periods. However, the protocol mentions only briefly when the topic was nominated and when it was presented to the NTP Board of Scientific Counselors (NTP 2017, p. 5). Some additional detail is provided in the section “Nominations to NTP” (NTP 2017, p. 2) and in the monograph itself, which acknowledges that the nomination was from the Fluoride Action Network, but the protocol does not address the extent of topics as outlined in the OHAT handbook.
• Problem formulation. According to the OHAT handbook, this section should “describe and document major decisions made during scoping and problem formulation. It should also describe how key scientific issues will be addressed in the evaluation. Problem formulation activities include discussions of the evaluation design team, preparation of scoping reports and any external activities, such as concept review by the NTP Board of Scientific Counselors, public comment, or webinars, listening sessions, or workshops undertaken to solicit input on specific scientific or technical issues” (NTP 2019a, p. 16). However, that information appears to be missing from the protocol. In the monograph, the section “Problem Formulation and Protocol Development” contains a list of problem-formulation steps, including input from the NTP Board of Scientific Counselors and a review of the draft protocol by technical advisers (NTP 2019c, p. 5). Notably missing is a discussion of opportunities for public engagement and comment except for acknowledgment that the protocol has been publicly available on NTP’s Web site since June 2017. The OHAT Web site appears to indicate that there were several public-comment periods in 2015 and 2016, but they are not reported or discussed in the protocol.

• Development of PECO statement. The OHAT handbook includes a section “Key Questions and Analytical Framework” that guides development of the PECO statement, but a similar section is not included in the protocol or the monograph.
• Screening and data extraction. In the protocol, four NIEHS staff and numerous ICF contractors are identified as involved in the screening step. The OHAT handbook states that if a contractor is used, a second reviewer should be an NTP staff member. It is unclear whether that guidance was followed. For data extraction, the protocol does not mention training for data extractors or pilot testing of all team members as recommended in the OHAT handbook.

To increase transparency, NTP should clearly describe the role of the OHAT handbook in developing the systematic-review protocol primarily to set expectations for how closely the process described in the handbook will be followed in the protocol and eventually the systematic review. That would help to address concerns about information that appears to be missing from the protocol or about conflicts between the protocol and the handbook.

Important Details

The presentation by NTP OHAT staff to the committee on November 6, 2019, indicated that the protocol for the monograph is intended to serve as a standalone document—that is, the protocol should contain all details relevant to the conduct of the systematic review. That position might be due partly to the fact that the OHAT handbook will change, as it states: “the procedures are a living document with the expectation that approaches will be updated as methodological practices are refined and strategies identified that improve the ease and efficiency of conducting a systematic review” (NTP 2019a, p. v). The OHAT handbook has already undergone one recent revision (NTP 2019b) and will likely undergo several more. Thus, it might be best if each systematic-review protocol could stand alone as an independent document that contains all the information necessary for understanding of the planning and conduct of the review. The committee, however, acknowledges that it would also be satisfactory to cite the appropriate OHAT handbook versions in the protocol for sections in which the details regarding the process align with the handbook to limit the need to repeat information from the handbook in the protocol.

Any details pertaining to the conduct of a review should be planned beforehand and described in the protocol. The committee found that many important details were missing from the protocol, although some of the information was contained in the OHAT handbook or the monograph. The committee recommends that the details be included in the protocol for transparency. Examples of important details missing in the protocol are as follows:

• Updating animal literature search. One of the specific aims of the systematic review was to update the experimental animal literature cited in NTP’s systematic review of the animal evidence on effects of fluoride on learning and memory (NTP 2016). However, details on the procedures used to update the literature search were lacking. The protocol provides database search strategies in Appendix 1, but these are not specific to the evidence stream and list only an end date for the search (December 19, 2016, for PubMed). The search strings also appear to differ from those used in the previous animal systematic review (NTP 2016). Thus, it appears that the literature search for animal studies is not a “re-execution”⁵ of the original search but rather an “update”⁶ of the search. However, it is unclear how the updated search specifically for animal studies was conducted and whether the modifications in search strategy resulted in the identification of new studies published before 2016. The monograph discusses the search strategies to some extent by stating that “literature searches for this systematic review were conducted independent of the literature search conducted for the NTP (2016) report using a similar strategy. As relevant animal studies published prior to 2015 were identified in the NTP (2016) assessment, the focus of the literature searches for this systematic review was to identify relevant animal studies that were published since completion of the literature searches for the NTP (2016) assessment” (NTP 2019c, p. 8). The specific procedure for updating the literature search for animal studies should be transparently outlined in the protocol with sufficient details to allow independent reproduction of the search.
• Evidence selection criteria. The protocol includes a detailed PECO statement but does not include explicit inclusion and exclusion criteria. That omission is critical because, although a PECO statement forms the basis of the criteria, detailed criteria offer greater clarity for understanding and documenting the screening process for identifying relevant studies. The criteria also increase transparency for potential reproduction of the review or facilitate updates to incorporate newer information. The example provided in the OHAT handbook illustrates how detailed inclusion and exclusion criteria provide greater clarity than simply a PECO statement alone (NTP 2019a. pp. 13, 17).
• Screening for inclusion. Studies were screened for inclusion by using a structured form in SWIFT-Active Screener, a machine-learning software program used to rank studies for screening. The National Academies has stated that automated screening procedures can facilitate efficiencies in the process and that incorporation of software tools, such as SWIFT-Active Screener, can help to achieve that goal (NRC 2014; NASEM 2018). However, those tools are relatively new and have not undergone rigorous evaluation or validation. Specifically, to the committee’s knowledge, they have not been validated for screening studies for inclusion in systematic reviews. Furthermore, screening up to 98% inclusion means that as many as 2% of the 13,023 studies excluded on the basis of the

___________________

⁵ See https://handbook-5-1.cochrane.org/chapter_3/3_4_2_1_re_executing_the_search.htm.

⁶ See https://handbook-5-1.cochrane.org/chapter_6/6_4_12_updating_searches.htm.

• SWIFT algorithm in this systematic review—260 studies—could be relevant according to title and abstract screening but missed in the initial screening. Given the large number of studies screened for this systematic review, that is not an insignificant number, although the committee notes that not all the studies would likely be deemed relevant in the full-text screening step. The OHAT handbook mentions the SWIFT text-mining and machine-learning tools but does not justify or cite why 98% estimated recall is considered sufficient. The committee recommends that the protocol discuss the basis of that decision and potentially conduct a sensitivity analysis to determine the effect of that cutoff on the overall findings (for example, by reviewing a random subset of the studies excluded on the basis of the SWIFT algorithm to identify the number of potentially missed references).
• Reporting excluded studies. It was decided in a May 2019 protocol revision not to list excluded studies because the number of studies excluded by using SWIFT Active during title and abstract screening was large. The OHAT handbook indicates that the list of included and excluded studies should be posted on the project’s Web site when screening has been completed to provide an opportunity for public review of the literature considered for evaluation. There is no mention of a size-cutoff criterion. The committee finds that further justification of the decision is warranted; this pertains to a list of 9,667 references screened and excluded and 13,023 references not screened—not an unreasonable number to present. In particular, the list of 2% unscreened studies based on 98% recall in SWIFT Active would be particularly appropriate to include because there is a chance that up to 260 studies could be missed as noted above.
• Screening and data extraction. The protocol lacks details relevant to screening and data extraction. For example, the protocol (NTP 2017, p. 29) describes the ideal evaluation team by stating that the “team members should have at least a master’s degree or equivalent level of experience in epidemiology, toxicology, environmental health sciences, or a related field.” However, it is unclear whether that criterion was met by the members of the review team inasmuch as only their names and affiliations are provided. Furthermore, more ICF contractors are listed in the monograph than in the protocol, but it is unclear when and why they were added. The committee recommends that the expertise and experience of all team members be provided for transparency to ensure that the review team has been established with expertise and experience appropriate for conducting the systematic review and recommends that OHAT guidance regarding the screening and data extraction process be followed.
• Data synthesis. The protocol does not include details about the planned conduct of statistical analyses, for example, models for meta-analyses, meta-regression, sensitivity analyses, or statistical evaluation of publication bias. The protocol contains only a section that discusses consideration for pursuing a narrative or quantitative evidence synthesis and addresses heterogeneity in the available evidence. One assumes that the approach outlined in the OHAT handbook was applied, but that is not explicitly stated. Given that data analyses can vary with

• specific study questions, it would be appropriate to include a section on data analysis in the protocol. That approach is consistent with Cochrane and other protocols for environmental health.
• Rating evidence. The protocol contains details for each factor that contributes to increasing or decreasing confidence in the body of evidence (see Table 4 “Key Factors when Considering Whether to Downgrade or Upgrade across a Body of Evidence,” NTP 2017, p. 14ff), but the committee did not find that these descriptions were sufficient to ensure reproducibility or transparency of the process. For example, the downgrade factor of “risk of bias” includes only a list of the critical factors that potentially contribute to high overall risk-of-bias ratings. However, that guidance does not sufficiently outline the criteria that make it appropriate to downgrade for risk-of-bias concerns. NTP should clearly define each factor, including key considerations that warrant upgrading or downgrading the body of evidence. If the factors for upgrading and downgrading the body of evidence align with criteria provided in the OHAT handbook, that should be explicitly stated, and the appropriate version of the handbook should be cited to indicate where additional details might be found.

Consistency

Some details outlined in the protocol appeared inconsistent with methods ultimately implemented in the monograph. Given the critical role of the protocol in the design and implementation of a systematic review, those potential discrepancies were concerning. Changes in the protocol are common, as illustrated by several transparently documented modifications of the protocol, but inconsistencies between the protocol and the monograph raise concerns regarding the rationale for specific changes and why they were not documented in the protocol.

• Consideration of mechanistic data. The protocol states that if “mechanistic data fail to provide support for biological plausibility of the relationship between exposure and the health effect, the hazard identification conclusion may be downgraded…from that initially derived” (NTP 2017, p. 22). However, the monograph differs by stating that “if mechanistic data provide strong opposition for biological plausibility of the relationship between exposure and the health effect, the hazard identification conclusion may be downgraded…from that initially derived” (NTP 2019c, p. 16). The committee finds that the latter approach is more appropriate because failing to provide support of a relationship is fundamentally different from providing support against a relationship. The committee concludes that there must be sufficient mechanistic evidence to warrant downgrading a hazard conclusion, not simply lack of evidence that supports a relationship.
• Outcome assessment. The protocol does not specify any subgroup or sensitivity analyses to be conducted. In the monograph, however, the decision to evaluate child and adult outcomes separately is presented. The protocol did not explicitly discuss or justify that decision or provide a definition to guide analysis.

• Confounders adjustment. In the protocol, the two critical confounders were identified as the potential for co-exposures (arsenic and lead) and iodine sufficiency. In the monograph, however, iodine deficiency or excess is listed as a potential confounding variable that might be considered important but not necessary; this is a major difference from the protocol, and it is unclear why it changed. Figure 6 in the monograph identifies key confounders as age, sex, arsenic, and socioeconomic status. Again, that is a discrepancy from what is discussed earlier in the monograph and in the protocol. Furthermore, a checkmark in Figure 6 indicates that the factor was considered and might have been adjusted for in the final model, but the criterion in the protocol for a low rating is that the study must provide quantitative summaries of the covariate and adjust for it in the analysis. Such discrepancies should be reconciled.

DATA PRESENTATION

As stated in Chapter 1, the committee did not conduct a data audit, but it did find some minor issues with data presented in the monograph. Some of the issues appear to be minor errors; for example, the number of references is incorrectly reported in Figure 4 “Study Selection Diagram” (NTP 2019c, p. 18). During the title and abstract screening, 9,667 references were screened and excluded; 13,023 references were not screened, on the basis of the SWIFT algorithm; and 807 references were included for full text review. Those numbers sum to 23,497—30 more studies than reported as screened in the figure (23,467).

Other presentation issues are related to enhancing the utility of the data presentation. For example, most tables and figures in the monograph are organized alphabetically by study author last name. That approach does not convey the information in a meaningful format. It might be better to organize the studies in a more informative way; for example, risk-of-bias tables could be organized by risk-of-bias ratings (that is, studies that have the most green “ratings” first and studies that have fewer such ratings thereafter) or by stratifying the studies according to critical risk-of-bias domains similar to the example provided in the OHAT handbook (NTP 2015, p. 39). Because critical domains⁷ might cause a

___________________

⁷ Key domains for humans include confounding, exposure characterization, and outcome assessment, and key domains for animals include randomization, exposure characterization, outcome assessment, and litter effects for developmental studies.

study to be excluded from the analysis, highlighting them in each table would help readers to interpret the overall risk-of-bias ratings. As another example, Table 6, “Studies on Neurodevelopmental and Cognitive Function in Humans,” also arranged studies alphabetically by study author last name (NTP 2019c, p. 22ff). An alternative suggestion would be to categorize studies by age cutoffs because the outcomes are evaluated separately for children and adults. Furthermore, NTP should include the detection limit of exposure measures for each study in Table 6. Organization of data is discussed further in Chapters 3 and 4.

COMMUNICATION

The committee noted that the monograph might benefit from improved communication in several respects. First, many people are interested in the question of whether water fluoridation to prevent dental decay poses a threat to cognition and neurodevelopment. Although the monograph includes some discussion of dose–response relationships, NTP did not conduct a formal dose–response assessment that could inform a discussion on water fluoridation. NTP needs to state clearly that the monograph is not designed to be informative with respect to decisions about the concentrations of fluoride that are used for water fluoridation. That point should be reiterated at the end of the monograph with some indication that its evaluation of the literature is focused on hazard identification of fluoride and that it does not draw any conclusions regarding drinking-water fluoridation or other fluoride sources, such as toothpaste or other dental treatments. Although NTP does not explicitly claim that it has done something other than hazard identification, the context into which the monograph falls calls for much more carefully developed and articulated communication on this issue.

Second, the monograph lacks details on the process of evaluating confidence in the body of evidence. As discussed earlier, the protocol lacks sufficient definitions for level-of-evidence descriptors as they pertain to the specific study question addressed in the monograph. However, the ratings applied for each confidence factor (Table 7, NTP 2019c, p. 51) also lack any justification or discussion. For example, NTP’s discussion of the overall risk-of-bias rating (NTP 2019c, p. 28) states that the confidence rating was not downgraded because 20 studies had little or no risk-of-bias concerns. However, NTP also states that the remaining human studies had probably high or definitely high risk of bias for at least two key considerations (exposure characterization, outcome assessment, or confounding factors). Thus, the rationale for not downgrading risk of bias is not entirely clear. NTP should consider supplementing Table 7 with clear justification for each confidence factor rationale for and why upgrade and downgrade factors were not applied for any of the human evidence.

Finally, there is little discussion of the process for obtaining missing or additional information from study authors. The monograph states that authors of included studies “were queried by email to obtain missing information and responses received were used to update risk-of-bias ratings” (NTP 2019c, p. 12). The author responses and changes in risk-of-bias ratings were documented in the

HAWC program. Obtaining additional information from study authors is critical for systematic review because it can minimize the effect of reporting bias vs other aspects of bias related to the design and implementation of studies. However, the monograph lacks clear documentation of how many authors were contacted, which authors were contacted, how many responded, and how risk-of-bias ratings were updated generally. Although that is tracked in HAWC with the actual risk-of-bias ratings, it could be helpful to include overall summary statistics of this critical process for greater transparency in the monograph. At a minimum, risk-of-bias ratings or extracted data that are updated on the basis of information obtained from the study authors should be clearly indicated for the specific risk-of-bias ratings or data in all relevant tables and figures.

REFERENCES

NTP. 2017. Protocol for Systematic Review of Effects of Fluoride Exposure on Neurodevelopment.

NTP. 2019a. Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration. Research Triangle Park, NC: National Institute of Environmental Health Sciences.

NTP. 2019b. Updates and Clarification to the OHAT Approach for Systematic Review and Evidence Integration Research Triangle Park, NC: National Institutes of Health. Available: https://ntp.niehs.nih.gov/ntp/ohat/pubs/handbookclarificationmarch2019_508.pdf.

NTP. 2019c. Draft NTP Monograph on the Systematic Review of Fluoride Exposure and Neurodevelopmental and Cognitive Health Effects. Office of Health Assessment and Translation, Division of the NTP, National Institute of Environmental Health Sciences, National Institutes of Health, US Department of Health and Human Services.

Rennie D., and I. Chalmers. 2009. Assessing authority. JAMA 301(17):1819-1821.

Rooney A.A., A.L. Boyles, M.S. Wolfe, J.R. Bucher, and K.A. Thayer. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environmental Health Perspectives 122(7):711-718.

Woodruff T.J., and P. Sutton. 2011. Navigation Guide Work Group. An evidence-based medicine methodology to bridge the gap between clinical and environmental health sciences. Health Aff (Millwood) 30(5):931-937.

Woodruff T.J. and P. Sutton. 2014. The Navigation Guide systematic review methodology: A rigorous and transparent method for translating environmental health science into better health outcomes. Environmental Health Perspectives 122(10):1007-1014.