Appendix D

Evidence Review: Methods and Approach

This appendix describes the methodological approach the committee used to address the portions of the statement of task that asked the committee to “review current scientific evidence to determine potential long-term health risks of exposure to JP-5” and “determine the strength of evidence for the spectrum of putative health effects following the Red Hill exposure, based on the review of current evidence” (see Box 1-1 in Chapter 1 for the full statement of task). The statement of task specifically asked the committee to do the following:

The committee decided that this portion of the statement of task required two different determinations: (1) qualitative categories that describe the strength of evidence of jet propellant 5’s (JP-5’s) putative health effects that can be used to prioritize clinical surveillance or monitoring, and (2) evidence gaps that contribute to uncertainty about health effects of most concern.

To produce these outputs, the committee developed a multistage process. The first stage aimed at cataloging what is known about JP-5 and its health effects. The committee identified all authoritative reviews of JP-5 (see Table D-1) and the known human health outcomes. The second stage was to identify any recent, high-quality systematic reviews investigating JP-5 and human health outcomes. To ensure all relevant literature was captured, a formal search was conducted by the National Academies Research Center. The committee then reviewed the literature cited by the authoritative and systematic reviews. During the second stage, literature captured in the review that did not meet inclusion criteria (i.e., not a systematic review or not human data) but remained relevant to the report were excluded and tagged for further review during the fourth stage. During the third stage, the committee identified all published literature assessing the exposed Red Hill population. This included analyses of surveys conducted within the affected population and medical chart reviews. Finally, during the fourth stage, the committee reviewed any remaining published research articles sourced from the authoritative and systematic reviews identified in stages one and two of the process that described the association between exposure to JP-5 and toxicological outcomes, and the effects of exposure to other hydrocarbon mixtures (e.g., JP-8, kerosene, JP-4).

To supplement the lack of recent, high-quality authoritative and systematic reviews, the committee then categorized the toxicity of JP-5 and similar hydrocarbon mixtures. These findings were then synthesized to inform the committee’s conclusions on the potential health outcomes following exposure to JP-5 or similar hydrocarbon mixtures.

The following sections of the appendix cover the committee’s analysis of the authoritative reviews, systematic reviews, and research on the exposed Red Hill population of interest. The final section covers the committee’s strength-of-evidence determination.

AUTHORITATIVE REVIEWS

The committee defined authoritative reviews as those produced by government agencies or other bodies that publish peer-reviewed strength-of-evidence determinations. The committee focused on national or international organizations and agencies that influence other organizations.

National and international organizations were also reviewed for supporting evidence on toxicology of components.

The following organizations met the criteria for authoritative reviews:

• Agency for Toxic Substances and Disease Registry (ATSDR)
• International Agency for Research on Cancer (IARC)
• National Toxicology Program (NTP)

• Department of Veterans Affairs (VA)
• American Petroleum Institute (API)

The search for authoritative reviews also included the Centers for Disease Control and Prevention (CDC), Organisation for Economic Cooperation and Development, and the European Chemicals Agency.

Step one aimed to identify authoritative reviews on the human health effects of JP-5. Because of the lack of available authoritative reviews, the committee refined its approach to identify authoritative reviews that focus on jet fuels generally. Finally, the committee conducted a search for authoritative reviews on the human health effects of four jet fuels with common occupational exposures and comparable chemical makeups: Jet A, JP-4, JP-5, and JP-8. Table D-1 summarizes the authoritative reviews identified by the committee.

The committee followed a multistep process to identify authoritative reviews. Among them, the ATSDR’s Toxicological Profile for JP-5, JP-8, and Jet A Fuels and the VA’s congressionally mandated report Health

TABLE D-1 Authoritative Reviews Assessed in Analysis

NOTES: API = American Petroleum Institute; ATSDR = Agency for Toxic Substances and Disease Registry; IARC = International Agency for Research on Cancer; JP-5 = jet propellant 5; JP-8 = jet propellant 8; NASEM = National Academies of Sciences, Engineering, and Medicine; NTP = National Toxicology Program; VA = Department of Veterans Affairs.

Effects of Jet Fuels Used by Armed Forces included the greatest number of studies relevant to the statement of task. Therefore, these two reviews were used as the basis of the proceeding evidence reviews.

The committee did not assess the quality of the authoritative reviews.

REVIEW OF SYSTEMATIC REVIEWS

The committee’s review of systematic reviews consisted of the following steps: conduct a literature search, screen relevant abstracts, conduct a full-text review of studies identified in the abstract screening, evaluate the final set of relevant studies. The committee then considered all the individual epidemiologic studies identified in each systematic review.

A secondary literature search was conducted following the identification of systematic reviews. This literature search was conducted to ensure all relevant literature was captured for the committee, including nonsystematic review literature with the study population of interest (i.e., individuals exposed during the Red Hill fuel releases).

Literature Search

The National Academies Research Center assisted the committee’s review of systematic reviews. Table D-2 outlines exclusion and inclusion search criteria. Systematic reviews were included regardless of publication date or country of origin. Systematic reviews were considered if they reviewed human studies, were published in English, and were classified as review papers. Furthermore, see Tables D-4, D-5, D-6, D-7, and D-8 for search engines and accompanying filters. The primary literature search was conducted between March 4, and March 31, 2025. The secondary literature search was conducted on April 11, 2025, and April 14, 2025. See Tables D-3, D-9, D-10, D-11, D-12, and D-13 for search engines and accompanying filters.

TABLE D-2 Literature Search #1 Inclusion and Exclusion Criteria

TABLE D-3 Literature Search #2 Inclusion and Exclusion Criteria

TABLE D-4 Medline Search Filters (Search #1)

TABLE D-5 Embase Search Filters (Search #1)

TABLE D-6 IEEE Filters (Search #1)

TABLE D-7 American Chemical Society Filters (Search #1)

TABLE D-8 Scopus Filters (Search #1)

TABLE D-9 Scopus Filters (Search #2)

TABLE D-10 MEDLINE Filters (Search #2)

TABLE D-11 Embase Filters (Search #2)

TABLE D-12 IEEE Filters (Search #2)

TABLE D-13 American Chemical Society Filters (Search #2)

Screening Abstracts

The literature search identified 353 potentially relevant systematic reviews. The publications were imported into PICO Portal, a web-based tool for collaborative citation screening for systematic reviews. Following the import into PICO Portal, 12 texts were identified as duplicates, leaving 341 texts to undergo screening by National Academies staff.

The abstract review used the following population, intervention, comparison, outcome, study (PICOS) statement:

Population: Humans exposed to jet fuels

Intervention: Oral, dermal, or inhalation exposure to kerosene-based hydrocarbon mixtures

Comparison: Any comparison group

Outcome: Any human health outcome; toxicity of jet fuels

Study: Systematic review papers (unless population is of Red Hill population or focuses on vulnerable groups)

The inclusion and exclusion criteria related to the PICOS statement were as follows:

Inclusion Criteria: Includes human evidence (with special attention to the Red Hill population and vulnerable groups); reviewed exposure to jet fuel, kerosene, and other hydrocarbon mixtures regardless of exposure pathway; assesses any health effect of outcome following exposure, including toxicity of components contributing to health effects; review paper has a methods section.

Exclusion Criteria: Assessed toxicological data or jet fuel emissions not pertaining to human health; singular exposure to gasoline or components of jet fuels other than kerosene; outcomes unrelated to physical or mental health; non-reviews of occupational exposures.

All title and abstract screening were conducted by at least two staff members, with adjudications being resolved by a predetermined project leader following a consensus conversation with other screeners.

Full-Text Review

Following abstract screening, 149 studies went on to full-text screening. The same two-person screening requirement was instituted for full-text reviews. Following full-text screening, 13 texts remained. Of the 13 texts, 8 pertained to the Red Hill population, leaving 5 systematic reviews for the risk-of-bias assessment.

The PICO Portal tag function was used during the screening process to flag literature relevant to the report but not meeting inclusion criteria (i.e., not epidemiological data or not a systematic review). This literature was then referenced during the fourth stage of evidence collection.

Evaluation and Evidence Assessment

The quality of the two included systematic reviews was evaluated using the risk of bias in systematic reviews (ROBIS) tool, which has been used by other National Academies committees. ROBIS includes several critical appraisal domains (see Box D-1). Quality assessment was conducted by two National Academies staff members and then adjudicated by a committee member. The committee conducted an evidence assessment because systematic reviews can be subject to a range of biases.

Synthesis: Results

The committee identified two systematic reviews that had clear methods for evidence identification, evidence evaluation, and synthesis (Carvajal et al., 2025; Vincent-Hall et al., 2025), and six narrative reviews (Fowles et al., 2016; Harris, 2010; Hoisington et al., 2024; Karanikas et al., 2021; Ritchie et al., 2001, 2003; Warner et al., 2015). The committee reviewed all the reviews with the ROBIS tool (Whiting et al., 2016). All narrative reviews were judged to have a high risk of bias and overall low confidence. The two systematic reviews had a low risk of bias and overall high confidence.

The two reviews (Carvajal et al., 2025; Vincent-Hall et al., 2025) judged to be low risk of bias were used as sources for epidemiologic studies reviewed by the committee, in addition to those in the Red Hill population. The two reviews were complementary in that the Vincent-Hall et al. (2025) review was primarily focused on occupational exposures to jet fuels, which includes both combusted and precombusted exposures, and the Carvajal et al. (2025) review was focused on precombustion forms of fuels.

Vincent-Hall and colleagues (2025) judged the risk of bias for each individual study using the Integrated Risk Information System (IRIS tool), a domain-based approach where each study is evaluated for risk of bias and study sensitivity; it is widely used to assess risk of bias in observational studies of chemical hazards and has been well accepted by a National Academies committee (NASEM, 2022).

Risk-of-bias domains include exposure and outcome assessment, confounding, population/selection, and selective reporting; together the domains inform an overall study confidence rating (high, medium, low, or uninformative). Qualifications for an overall confidence rating level are as follows (NASEM, 2022):

• High: No or one noncritical weakness: the systematic review provides an accurate and comprehensive summary of the results of the available studies that address the question of interest.
• Moderate: More than one noncritical weakness: the systematic review has more than one weakness but no critical flaws. It may provide an accurate summary of the results of the available studies that were included in the review.
• Low: One critical flaw with or without noncritical weaknesses: the review has a critical flaw and may not provide an accurate and comprehensive summary of the available studies that address the question of interest.
• Critically Low: More than one critical flaw with or without noncritical weaknesses: the review has more than one critical flaw and should not be relied on to provide an accurate and comprehensive summary of the available studies. (p. 229)

The IRIS handbook includes several critical appraisal domains (see Box D-2). The risk-of-bias assessment was conducted by two staff members and confirmed or adjudicated by a committee member. The committee conducted an evidence assessment because epidemiological studies can be subject to a range of biases.

Vincent-Hall and colleagues (2025) found that most studies were judged low confidence or uninformative—typically for deficient exposure measurement (potential exposure based on job title/duration proxies, little or no quantitative monitoring), potential confounding, and limitations in outcome ascertainment—and no study reached high confidence.

Carvajal and colleagues (2025) judged study quality using the Modified Downs and Black Checklist and the Joanna Briggs Checklist for Case Reports. The authors found that study quality varied by design: case reports scored highest on average (77 percent good)—strong on patient/presentation descriptions but weak on reporting unanticipated or harmful

posttreatment effects; cross-sectional studies were generally fair (49 percent), with common gaps in recruitment/demographics, confounder control, exposure assessment, and dose-response analysis; and the sole cohort study was good (71 percent). Of the three analytical studies, two were good/excellent (Fuente et al., 2019; Heaton et al., 2017) and one was poor. A quality assessment was not assessed for the descriptive ethnographic study (Bateganya and Nakanjako, 2023), the “hospital based descriptive study” (Subramanian et al., 2018), and the analytical case series (Reddy et al., 2020).

REVIEW OF LITERATURE IN THE RED HILL POPULATION

The committee communicated with the Navy, Red Hill Registry, and impacted community members to obtain studies assessing the exposed population. More specifically, the committee contacted the VA, Department of Defense (DoD), CDC, and Hawai‘i State Department of Health for a list of known research articles published following the Red Hill November 2021 fuel release. Community liaisons were notified of the request for published literature. The committee then worked with the National Academies Research Center to confirm all existing literature on the exposed Red Hill population had been identified, ensuring adequate search criteria

were included during the search for systematic reviews to capture Red Hill–related studies. The evidence presented below reflects the committee’s synthesis from Chapter 4.

As of October 30, 2025, eight epidemiologic studies or reports evaluating the physical and mental health outcomes following the Red Hill JP-5 drinking-water contamination were found: seven descriptive investigations, community surveys (Bremer et al., 2025; Miko et al., 2023; Troeschel et al., 2022), or medical records summaries (EDC, 2023a, 2023b; Nguyen et al., 2025; Saunders et al., 2025) and one retrospective cohort study with a comparison group (DHA, 2024). Troeschel and colleagues (2022) and Miko and colleagues (2023) both present data from the ATSDR Assessment of Chemical Exposures (ACE) survey, whereas Bremer and colleagues (2025) is a follow-on survey conducted by University of Hawai‘i Economic Research Organization (UHERO). The committee conducted a risk-of-bias assessment for each of the identified reports through the IRIS Handbook (see Box D-2).

The committee found that the studies of the Red Hill population were all methodologically flawed and had the potential for selection bias, exposure, and outcome misclassification bias. One study of the Red Hill population was judged to be of medium confidence (DHA, 2024), and all others were judged to be of low confidence.

Selection bias occurs when the study sample is not representative of the target population. The medical record reviews consisted only of patients who received care from DoD-related providers, which may have selected for a population that was healthier and more trusting of this military care compared with the level of trust among those receiving civilian care, and may be less likely to have reported health effects. The first ACE survey had a very low (14 percent) response rate (1,389 households participating out of 9,694 eligible), and the UHERO survey (Bremer et al., 2025) had an even lower response rate of 9.16 percent (174 respondents out of 1,900 eligible). These surveys may have led to selection bias if symptomatic people were more likely to participate. No study assessed exposure with an empirical measure. All studies had a risk of bias from exposure misclassification.

Furthermore, in all studies, exposure was assumed if a person lived in the affected areas at the time of the November 2021 release. In two of the medical record reviews, the reference period was prior to the November 2021 release (November 2020–November 2021) (baseline comparison group), compared with 1 year after November 2021 (November 2021–November 2022). Thus, the baseline comparison group could have included individuals who might have been exposed to JP-5 from the May 21, 2021, release, which could have led to a bias toward the null (Chapter 3). Outcome ascertainment was also flawed in all studies. In the surveys, all outcomes were self-reported (Bremer et al., 2025; Miko et al., 2023; Troeschel

et al., 2022). Miko and colleagues (2023) triangulated findings in the ACE survey with data from calls to the Hawai‘i Poison Center and the National Syndromic Surveillance Program, as well as interviews with key informants, which largely supported the ACE survey’s findings. The studies conducted by the Defense Health Agency (DHA) and the EpiData Center (EDC) relied on International Classification of Diseases 10 codes, which generally have high specificity but low sensitivity. Nondifferential misclassification biases affect estimates toward the null, although when misclassification differs by exposure group, the bias can be in either direction with respect to the null (Copeland et al., 1977; Goldstein et al., 2022).

ANIMAL AND MECHANISTIC STUDIES

Although the statement of task called for the committee to “assess the strength of evidence for the spectrum of putative health effects suggested by human studies,” the committee recognizes the importance of toxicological evidence in cases where little human epidemiological data are available. Therefore, the committee supplemented knowledge using the toxicological data included in authoritative reviews, such as the ATSDR’s (2017) Toxicological Profile for JP-5, JP-8, and Jet A Fuels; systematic reviews identified in the National Academies literature review; and other authoritative reviews that provide integrative conclusions based on multiple lines of evidence. In synthesizing evidence in this manner, the committee acknowledges that animal evidence greatly improves the interpretation of human studies, particularly for those health effects in which human epidemiological data are not yet available.

An observed association between JP-5 exposure and health effects does not necessarily mean that the exposure is the cause of that outcome. Toxicologic evidence, whether it supports or conflicts with epidemiological evidence, provides insights into biologic processes and informs how an observed association might be interpreted. The degree of biologic plausibility itself influences whether the committee perceives positive findings to be indicative of a pattern or the product of statistical fluctuations. Ultimately, the “results of toxicology studies should be consistent with what is known about the human disease process if they are to support a conclusion that the development of the disease was influenced by an exposure” (NASEM, 2018b, p. 59).

STRENGTH-OF-EVIDENCE DETERMINATION

To assess the strength of evidence regarding the potential for JP-5 to cause a particular health effect, the committee integrated the evidence reviewed in the ATSDR’s (2017) Toxicological Profile for JP-5, JP-8, and

Jet A Fuels and other authoritative reviews with the evidence from the systematic reviews and epidemiological studies.

The synthesis of available data was guided by a framework based on the Bradford Hill considerations (Hill, 1965), which help to determine whether associations are causal (see Box D-3). The committee did not consider the Bradford Hill considerations to be a heuristic for assessing causation in isolation—that is, as a checklist where each item must be met to establish causality. Rather, the committee considered them as a list of possible considerations meant to generate thoughtful discourse by the committee to help inform its determinations for the strength of evidence (Figure D-1) (Fedak et al., 2015; NASEM, 2018a).

In applying this framework, the committee assigned the greatest weight to studies conducted in the Red Hill population, given their direct relevance to the exposure scenario under review. Next in weight were the nonoccupational epidemiologic studies included in the Carvajal et al. (2025) review,

followed by the occupational studies summarized in the Vincent-Hall et al. (2024) review. The occupational literature was considered informative but was judged to carry lower weight overall, as most studies involved mixed exposures to both combusted and uncombusted fuels and relied on job title or duration proxies for exposure. The committee considered the animal studies discussed in the authoritative reviews when making its determinations, as an aid to the interpretation of human studies.

The committee did not derive separate conclusions based on route of exposure because the Red Hill community was exposed through ingestion, inhalation, and dermal contact. Similarly, the committee considered all kerosene-based jet fuels (e.g., JP-5, JP-8, Jet A) together in drawing conclusions, given their overlapping hydrocarbon ranges and similar toxicologic profiles.

Categories of Association

Sufficient Evidence of an Association

For effects in this category, a positive association between JP-5 and the outcome must be observed in studies in which chance, bias, and confounding can be ruled out with reasonable confidence. For example, the committee might regard sufficient evidence of an association as evidence from several small studies that is unlikely to be due to confounding or to be otherwise biased and that shows an association that is consistent in magnitude and direction. Experimental data supporting biologic plausibility strengthen the evidence of an association but are not a prerequisite,

nor are they sufficient to establish an association without corresponding epidemiologic findings.

Limited or Suggestive Evidence of an Association

In this category, the evidence must suggest an association between exposure to JP-5 and the outcome in studies of humans, but the evidence can be limited by an inability to rule out chance, bias, or confounding with confidence. One high-quality study may indicate a positive association, but the results of other studies of lower quality may be inconsistent.

Inadequate or Insufficient Evidence to Determine an Association

If there was not enough reliable scientific data to categorize the potential association with a health effect as sufficient evidence of an association, limited or suggestive evidence of an association, or on the other end of the spectrum, limited or suggestive evidence of no association, the health outcome was placed in the category of inadequate or insufficient evidence to determine an association by default. In this category, the available human studies may have inconsistent findings or be of insufficient quality, validity, consistency, or statistical power to support a conclusion regarding the presence of an association. Such studies may have failed to control for confounding factors or may have had inadequate assessment of exposure.

Limited or Suggestive Evidence of No Association

A conclusion of no association is inevitably limited to the conditions, exposures, and observation periods covered by the available studies, and the possibility of a small increase in risk related to the magnitude of exposure studied can never be excluded. However, a change in classification from inadequate or insufficient evidence of an association to limited or suggestive evidence of no association would require new studies that corrected for the methodologic problems of previous studies and that had samples large enough to limit the possible study results attributable to chance.

REFERENCES