5

Evidence Synthesis in Integrated Science Assessments

Chapter 4 details the types of individual studies that contribute to the body of evidence used in Integrated Science Assessments (ISAs). The weight of evidence approach articulated in the Preamble (EPA, 2015a) calls for the evidence to be synthesized and integrated across those studies, to arrive at the causal determination. This chapter describes evidence synthesis processes as articulated in the Preamble for first health effects and then welfare effects. Descriptions of procedures developed and implemented since the publication of the Preamble are also provided.

CURRENT APPROACH TO EVIDENCE SYNTHESIS AND INTEGRATION FOR HEALTH EFFECTS

Section 5 of the Preamble is titled “Evaluation, Synthesis, and Integration of Evidence across Disciplines and Development of Scientific Conclusions and Causal Determinations” (EPA, 2015a). The main types of studies used for both health and welfare outcomes are outlined; each of these study types has notable strengths and limitations, as discussed previously—the confidence of causal inference is strengthened when results are convergent across multiple study approaches and disciplines and weakened when they are not. The Preamble recognizes that experts are necessary and need to be included at many stages of preparing an ISA: during selection of policy-relevant questions in the IRP process; suggesting literature to consider; evaluating individual study quality; and evaluating preliminary draft assessments (EPA, 2015a). Thus, scientific expertise is a critical input to the weight of evidence approach throughout the ISA development process. The Preamble does not state how experts are selected but notes the need for a breadth of disciplines, itemized elsewhere in this chapter, and that the inclusion of the necessary disciplines may change with evolving methods and tools (EPA, 2015a). Experts are called on after ISAs are drafted through Clean Air Scientific Advisory Committee (CASAC) review.

The Preamble notes other causal determination frameworks such as those of the Institute of Medicine (IOM), the International Agency for Research on Cancer (IARC), and the Centers for Disease Control and Prevention (CDC) in the section called “Considerations in Developing Scientific Conclusions and Causal Determinations” (EPA, 2015a). Some of these frameworks are summarized

in Chapter 6 of this report. The basis for a weight of evidence assessment is the modified Bradford Hill aspects of association (Hill, 1965) (Table I of the Preamble). The Preamble notes that statistical significance, consistency of effects across several studies, quality of the studies, and other criteria may also be relevant in establishing causality. Finally, the Preamble notes the need to assess biological plausibility in addition to statistical conclusions, but states that a complete understanding of mode of action is not required (EPA, 2015a).

The criteria for the U.S. Environmental Protection Agency’s (EPA’s) five-level hierarchy of causal determinations (see Table 1.1) are conceptually the same for health and welfare, although different criteria are listed for health and welfare outcomes. The differences are more a reflection of different types of studies than different concepts of what constitutes causality. These differences include: the distinction between short-term and long-term effects; different types of endpoints; effects at different concentrations of air pollutants (the most relevant are those at or slightly above those experienced in the population; significantly higher exposures may give useful insights into biological mechanisms) and the need to assess concentration-response or dose-response curves to tie possible effects to observed exposures in various populations. All of these are considered in an integrated assessment. The implication throughout this discussion is that the same basic criteria are in operation for both health and welfare. While the Preamble provides a framework for making causal determinations, it is instructive to examine how the framework is applied in recent ISAs, including the 2019 PM ISA and 2020 Ozone ISA.

INTEGRATION AND SYNTHESIS EXAMPLES FOR HEALTH EFFECTS FROM THE 2019 ISA FOR PARTICULATE MATTER AND THE 2020 ISA FOR OZONE

Each of these recent ISAs starts with an executive summary of more than 20 pages, followed by a lengthy opening chapter that synthesizes of the entire report. This overview is followed by a series of chapters (called appendices in the Ozone USA) addressing specific aspects of the assessment. For example, the PM ISA (EPA, 2019c) has chapters on atmospheric chemistry, exposure assessment, and dosimetry, followed by individual health outcomes such as respiratory, cardiovascular, metabolic effects, and so on. Although the key points of the report are included in the synthesis chapter, it is necessary to read the individual chapters or appendices to understand how these assessments were reached. In this summary of ISA examples, the committee first describes the presentation of synthesis results, and then provides details about some of the individual assessments.

In the 2019 ISA for PM (EPA, 2019c), the synthesis chapter is highlighted by two tables that summarize the health effects of PM_2.5 (there are separate discussions for PM10-2.5 and ultrafine particles). In Table 1-1 health effects labeled as either causal or likely to be causal are listed. In much more detailed Table 1-2 all the causal conclusions are listed with the reasoning behind them. Quoting to give an example of how these conclusions are presented, the section of this table on “cancer and long-term PM_2.5 exposure” includes the following causal determination:

The conclusion from this summary is “likely to be causal,” which is a change from the previous ISA (EPA, 2009) that classified this relationship as only suggestive of a causal relationship, noting that the upgrade of the classification results from a decade of new research.¹

As a more detailed example of evidence integration and synthesis for a specific health endpoint, we summarize the evidence presented in Chapter 7 of the ISA for the long-term metabolic effects of PM_2.5, an emerging area of investigation. The ISA notes that while a small number of studies related to metabolic diseases were considered in the 2009 PM ISA (EPA, 2009), primarily for the potential of these conditions to increase susceptibility for cardiovascular effects, no causal determinations for metabolic effects were included in that assessment. The body of evidence related to metabolic effects of long-term exposure to PM_2.5 has increased over the past decade.

For long-term exposure to PM_2.5, the ISA chapter includes the outcomes of metabolic syndrome; glucose and insulin homeostasis; type 2 diabetes; other indicators of metabolic function including gestational diabetes; and diabetes-related mortality (EPA, 2019c). The discussion of biological plausibility includes pathways of systemic inflammation; inflammation of the hypothalamus; and autonomic nervous system (ANS) modulation. Animal toxicology studies involving diabetic mouse and genetic knockout models (e.g., Kampfrath et al., 2011; Liu et al., 2014) are discussed as having been significant in identifying and elucidating these pathways. The section reviews ten epidemiologic studies examining associations of long-term exposure with a metabolic syndrome diagnosis or indicators of glucose and insulin homeostasis (EPA, 2019c, Table 7-6) and eight animal toxicology studies related to these endpoints (EPA, 2019c, Table 7-7). The epidemiologic studies were critiqued as commonly lacking consideration of co-pollutant confounding, but the positive associations they found were assessed to be consistent with the animal toxicology results.

The ISA also reviews seven long-term epidemiologic studies for incident type 2 diabetes. Results are summarized in a forest plot (EPA, 2019c, Figure 7-10), which shows that the associations were generally positive though some relative risk ratios had wide confidence intervals that included the null. Effect modification by sex was also inconsistent between studies (EPA, 2019c). The section covers 13 toxicology studies of the influence of concentrated air pollution (CAP) exposure on inflammation and/or other indicators of metabolic function (EPA, 2019c, Table 7-9). These studies are found to support biological plausibility for the systemic inflammation pathway and for changes in liver function due to CAP exposure. The section then discusses results from three epidemiologic studies of metabolic disease that adjusted for co-pollutant confounding, which in two cases rendered the PM_2.5 effect insignificant. Lastly, epidemiologic studies of long-term PM_2.5 exposure and mortality from metabolic disease are considered as evidence of “an overall continuum of effects.” (EPA, 2019c, p. 7-47). Three studies by different groups using the same dataset with different exposure estimation methods found positive associations with similar hazard ratios (Jerrett et al., 2017; Pope et al., 2015; Turner et al., 2016). Multiple studies using a large Canadian cohort similarly found consistent positive associations (Brook et al., 2013; Crouse et al., 2015, 2016).

To support the determination that the evidence is suggestive of a causal relationship, the concluding summary table (EPA, 2019c, p. 7-13) lists (1) consistent findings in epidemiologic studies of diabetes-related mortality at relevant concentrations; (2) inconsistent findings of association with incidence of type 2 diabetes but with some high quality studies showing positive associations; (3) consistent positive associations for metabolic syndrome; (4) lack of consideration of co-pollutant

___________________

¹ The International Agency for Research on Cancer (IARC) has classified air pollution in general, and PM separately, as Class 1 (causal) carcinogens.

confounding in most of the epidemiologic studies; (5) uncertainty about influence of measurement error due to the limited database; (6) “strong” toxicologic evidence for some mechanistic pathways but questions of robustness as multiple studies were conducted using CAPs from the same location; (7) inconsistent findings regarding gestational diabetes; and (8) general coherence between toxicologic evidence and positive associations with metabolic disease mortality, diabetes, and cardiovascular disease (EPA, 2019c). It is not clear from the discussion how the countervailing items in this list were weighed, although emphasis is placed on the point that co-pollutant confounding could not be ruled out in most of the epidemiologic studies of metabolic disease. It is also not fully clear in the ISA how the individual studies were selected for inclusion, or which studies were excluded and why.

INTEGRATION AND SYNTHESIS EXAMPLES FOR HEALTH EFFECTS FROM THE 2020 OZONE ISA

The 2020 Ozone ISA (EPA, 2020a) is similarly organized to the 2019 PM ISA (EPA, 2019c), with a 100-page chapter on “Integrated Synthesis,” followed by a series of appendices similar to the chapters of the PM ISA (atmospheric chemistry, exposure, human health effects, etc.); the treatment of “welfare effects” (primarily ecological and climatic) is more detailed than in the PM ISA because there is no separate ISA for ecological effects of ozone.

Early sections of the “Integrated Synthesis” chapter cover atmospheric chemistry, ozone measurements and trends, and exposure (both human and ecological). The substantive human health assessment is the subject of Section IS-4, which covers numerous health effects of ozone (respiratory, metabolic, cardiovascular, total mortality, nervous system, cancer, reproductive effects) under both short-term and long-term ozone exposure. There is also a section aimed at identifying at-risk populations. The methods rely on three types of studies (as in the PM ISA): controlled human exposure studies, epidemiology, and animal toxicology, the latter being particularly cited to justify the “biological plausibility” part of the assessment. The results are presented in a series of tables, where the 2020 causal judgments are compared with those from the previous ozone ISA (EPA, 2013a).

Section IS-6 is titled Key Aspects of Health and Welfare Effects Evidence. This section summarizes the criteria that EPA used in making assessments. The Bradford Hill aspects of association (Hill, 1965) are again cited as the basis of causal assessments. Uncertainty is also a key factor, though the report emphasizes that “some conclusions may be robust to … uncertainties” and concludes, “Where there is clear evidence linking ozone with health and welfare effects with or despite minimal remaining uncertainties, the U.S. EPA makes a determination of a causal or likely to be causal relationship” (EPA, 2020a, p. IS-82). Detailed summaries of the process used for each of the health and welfare causal determinations follow.

As an example of how the 2020 Ozone ISA (EPA, 2020a) reached its conclusions, this paragraph summarizes Appendix 6 on the effect of both short-term and long-term exposure to ozone on total (nonaccidental) mortality. In both cases, the conclusion was that ozone exposure was “suggestive of, but insufficient to infer, a causal relationship.” The conclusion for long-term ozone exposure is consistent with that of the previous 2013 ozone ISA (EPA, 2013a); however, the conclusion for short-term ozone exposure represents a down weighting compared with the 2013 conclusion “likely to be a causal relationship.” Here, we summarize the reasoning that EPA gives for those conclusions.

This section begins with an outline of the PECOS tool, newly introduced in this ISA, that uses population, exposure, comparison, outcome, and study design as the five criteria for selecting papers to review. All five criteria must be satisfied for a study to be included in the initial review.

However, the authors are still not explicit about the basis on which some studies are included, and others excluded, under these criteria.

Referring first to short-term ozone exposure, a short section on “Biological Plausibility” details animal toxicology, controlled human exposure, and epidemiological studies, noting that “there is limited evidence of relationships between short-term ozone exposure and more severe cardiovascular effects, such as ED visits and hospital admissions” (EPA, 2020a, p. IS-44) and concludes that “this lack of coherence in results between experimental and epidemiologic studies contributes to uncertainty in the observed associations between short-term ozone exposure and mortality because the contribution of cardiovascular mortality to total mortality is relatively large” (EPA, 2020a, p. IS-44). Subsequent sections consider epidemiological evidence for the influence of short-term ozone exposure on mortality, mostly based on multicity or multicontinent studies. There are sections on total nonaccidental mortality, cause-specific (cardiovascular or respiratory) mortality, possible effect modifiers, confounding by co-pollutants, weather, and season, and the shape of the concentration-response curve (“no evidence of a threshold,” but noting uncertainty in the shape of the curve at low ozone levels). The discussion of long-term ozone exposure and mortality follows similar lines, but with less evidence overall for a causal effect. The ISA states “the available evidence for respiratory morbidity and metabolic disease supports potential biological pathways by which long-term ozone exposures could result in mortality; however, for cardiovascular morbidity, the evidence is more limited” (EPA, 2020a, Appendix 6, p. 27). The review of epidemiological evidence notes overall inconsistent studies, with the strongest evidence for an ozone-mortality association coming from analysis of a Medicare cohort (Di et al., 2017b; Zanobetti and Schwartz, 2011). The concluding Table 6.2 records “limited and sometimes inconsistent epidemiologic evidence from multiple, high-quality studies at relevant ozone concentrations” (EPA, 2020a).

The remainder of the Appendix includes tables (6-3 through 6-9) listing all the individual studies included and summarizing their main results, and concludes with an “Annex” headed “Scientific considerations for evaluating the strength of inference from studies on the health effects of ozone” (EPA, 2020a). The annex lists the study quality criteria used to determine the selection of studies for inclusion in the ISA, and the previously mentioned PECOS tools were used to help determine study relevance. However, there is no indication of EPA’s conclusions about the relative quality of the individual studies listed in Tables IS6-3 through IS6-9, nor why some other studies reporting indeterminate or negative effects of short-term ozone on mortality (e.g., Smith et al., 2009; Young et al., 2017) were not included at all. Smith et al., 2009 was referenced in the 2013 ozone ISA, but not in the 2020 ISA. As indicated in EPA’s Health and Environmental Research Online (HERO) database,²Young et al. (2017) was included in the 2019 PM ISA (and so presumably passed study quality and relevance screening there) and was “considered” for the 2020 ozone ISA, but was excluded from that ISA during the full text screening process for reasons that aren’t readily accessible in HERO or the ISA.

CURRENT APPROACH TO EVIDENCE SYNTHESIS AND INTEGRATION FOR WELFARE EFFECTS

The Preamble’s causal determination framework uses modified Bradford Hill aspects of association (Hill, 1965) aspects for both health and welfare endpoints (EPA, 2015a, Table 3.1), although the types of evidence and the relative weighting of lines of evidence are somewhat different (EPA, 2015a). EPA also summarizes information on welfare endpoints using the same five descriptors of causal determinations as are used for public health effects (EPA, 2015a, Table 3.2). Causal assessments for welfare include relatively broad outcome categories or groups of related endpoints, such

___________________

² See https://hero.epa.gov/hero/index.cfm/content/home (accessed May 26, 2022).

as reduced crop yields or increased tree mortality. To make causal determinations, EPA assesses consistency and the overall strengths and limitations of the multidisciplinary body of evidence, drawing from all studies judged to be of adequate quality and relevance. Determinations are supported by tabular summaries of the evidence and narrative explanations. EPA’s 2015 Preamble to the ISAs states that the modified Bradford Hill aspects of association are considerations, but not to be applied as a checklist; not meeting one doesn’t preclude a determination that a relationship is causal (EPA, 2015a). Consistent with the Framework, as stated in the NO_x-SO_x-PM Integrated Review Plan, for ecological endpoints, “key considerations in drawing conclusions about causality include consistency of findings for an endpoint across studies, biological plausibility, and coherence of the evidence across disciplines and across related endpoints” (EPA, 2017a).

As discussed in Chapter 4, types of evidence considered for ecological welfare endpoints include controlled experiments in laboratory or greenhouse settings, open top field chambers, chamberless fumigation studies, field plot and whole-ecosystem manipulation experiments, time-series and paleoecological investigations and observational studies conducted across gradients of exposure. Atmospheric chemistry and transport models, coupled atmospheric chemistry-climate models and coupled atmosphere-land-water process models contribute important evidence for causal determinations for effects of criteria pollutants on ecosystems, visibility, radiative forcing, and climate.

INTEGRATION AND SYNTHESIS EXAMPLES FOR WELFARE FROM THE 2020 OZONE ISA

For ecological effects in the 2020 ozone ISA, causal determinations are presented for 12 major outcome categories (EPA, 2020a, p. 8-1). Aiding in the assessment of coherence, the categories are ordered from the lowest to the highest level of complexity in biological organization, starting with visible foliar injury and including reduced quality or yield of agricultural crops, increased tree mortality, and alteration of terrestrial community composition. Exposure levels of interest for ecological effects in the ISA are specified in advance in the PECOS criteria (EPA, 2020a, Table 8-2).

One issue with ecological effects for ozone is the disconnect between the form of the current secondary standard, which matches the primary standard in using the fourth highest daily maximum 8-hour-average concentration, averaged over 3 years. In contrast to this short-term, 8-hour, average exposure metric, most vegetation research is based on cumulative exposure over the growing season. In the ozone ISA (EPA, 2020a) this incongruity is noted but a detailed explanation is lacking, rather the 2013 ozone ISA (EPA, 2013a) is referenced for further discussion about the preferred cumulative exposure metrics. Tables in the 2020 ISA summarizing individual studies identify the exposure metric, level, and pattern of exposure they evaluated (EPA, 2020a). However, statements in the ISA summarizing the causal determinations do not specify the exposure metric or level for which causality is being determined.

For all ecological outcomes in the ozone ISA (EPA, 2020a), the explanations of how evidence is synthesized to reach the causal determinations are relatively brief. This may in part reflect the judgment of EPA that the evidence for causality is clear, especially when many controlled experiment studies support the conclusion. A case in point, using evidence synthesis for reaching a strong conclusion regarding causality is the determination that ozone exposure causes visible foliar leaf injury. The supporting evidence for this determination includes Bergmann et al. (2017) comprehensive literature review of experimental studies, which found that 245 plant species (more than half of those tested) from 28 plant genera have been demonstrated to suffer foliar injury from ozone exposure in field experiments or gradient studies (EPA, 2020a). The comprehensive literature review, together with other studies and expert peer review, led to the determination of causal relationship for visible foliar injury.

The outcome of increased tree mortality from ozone exposure provides an example of a more involved determination drawing on multiple lines of evidence. No causal determination was made for the tree mortality outcome in the 2013 ISA, although the review included studies of this endpoint (EPA, 2013a). The 2020 ISA (EPA, 2020a) cites several new studies that have been added to the literature since then, especially highlighting a large multivariate analysis of tree mortality data from the U.S. Forest Service’s Forest Inventory and Analysis (FIA) database (Dietze and Moorcroft, 2011). This study applied regression analysis with climate, air pollutant, topographic and stand variables for approximately 3.4 million tree and 750,000 plot measurements taken between 1971–2005 in the central and eastern United States. The ISA also cites other observational studies that demonstrated the co-occurrence of tree mortality with elevated ozone but were more limited in their ability to rule out other contributing factors, as well as fumigation experiments that found increased mortality in sensitive tree genotypes. Finding that observations of increased tree mortality are consistent with well-established mechanisms explaining ozone phytotoxicity, the ISA concludes that ozone is likely to be causal for this endpoint (EPA, 2020a).

The 2020 ozone ISA (EPA, 2020a) contains a separate chapter on the effects of ozone on climate change, considering two outcomes: radiative forcing and “temperature, precipitation, and related climate variables.” EPA concludes that ozone causes radiative forcing and is likely causal for changes in temperature, precipitation, and related climate variables. The climate chapter is quite brief because for this topic the ISA relies on findings from the comprehensive review in the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC, 2013). Based on IPCC findings, the ISA concludes that a causal role for ozone in radiative forcing is supported by a well-established physical mechanism and by multiple studies with global chemistry-climate models that adequately reproduce global temperature trends. EPA states that the relationship between ozone levels and changes in temperature, precipitation, and other climate variables is complicated by the heterogeneous distribution of ozone in the atmosphere, by uncertainty about its pre-industrial levels and distribution, and by feedbacks and interactions within the climate system. The ISA concludes that ozone is “likely causal” for temperature and precipitation effects. The explanation for this determination focuses on uncertainty about the “precise magnitude of climate responses to ozone changes, particularly at regional scales” along with limited modeling studies for precipitation effects (EPA, 2020a, p. 9-22). This determination appears to conflate the question of causality with questions of effect quantification. Elsewhere in the ISA these questions are more clearly separated.

INTEGRATION AND SYNTHESIS EXAMPLES FROM THE 2020 NO_x-SO_x-PM ISA

The 2020 NO_x-SO_x-PM ISA (EPA, 2020b) considers whether concentrations or deposition of sulfur and nitrogen compounds cause effects through acidification, nitrogen enrichment, and sulfur enrichment. Causal assessments are conducted separately for terrestrial, wetland, freshwater and estuarine/near-coastal ecosystems. As an example of the classes of effects used in the NO_xSO_x-PM ISA, the grouping “N and acidifying deposition to terrestrial ecosystems” is divided into three linked subcategories: (1) alteration of soil biogeochemistry; (2) alteration of physiology and growth of terrestrial organisms and ecosystem productivity; and (3) alteration of species richness, community composition, and biodiversity (EPA, 2020b, Table ES-1). For PM, the ISA considers impacts on radiative flux that may alter photosynthesis, as well as direct and indirect effects of deposition of metals and organics on plants and soil communities. Of the 18 effects categories for which causal determinations are summarized in the 2020 ISA, 17 are classified as “causal” and one (“other ecological effects of PM”) as “likely causal” (EPA, 2020b, Table ES-1).

For the NO_x-SO_x-PM ISA studies conducted with concentrations or deposition loads that are within two orders of magnitude of current conditions were eligible for inclusion in the review

(EPA, 2020b, p. ES-7). The review includes studies conducted internationally but emphasizes those conducted in North America or relevant to species or ecosystem types found in North America. The ISA discusses critical loads for N and S deposition and presents maps of critical loads for multiple endpoints. While the ISA expressly disclaims the task of quantifying the risk that critical loads are or will be exceeded (EPA, 2020b, IS-3), studies that quantify critical loads are important for synthesizing experimental and observational evidence.

Appendix 6 of the NO_x-SO_x-PM ISA (EPA, 2020b), which covers nitrogen enrichment effects for terrestrial ecosystems, provides an illustration of how evidence is integrated in assessing causality. The Appendix reviews experimental and mechanistic studies of changes in the physiology of organisms, N-addition and observational studies of the relative growth or mortality of different species, larger-scale community composition and forest inventory surveys, and process models that synthesize response estimates and can be evaluated against historical trends over extended time periods. Tabular and graphical summaries of individual studies are used to synthesize results across studies and assess coherence across study locations, vegetation types, and study designs (e.g., EPA, 2020b, Figures 6-2 and 6-3). Several recent meta-analyses are highlighted in the ISA that have synthesized information from dozens of separate N-addition studies of effects on plant biomass growth, ecosystem productivity and carbon storage, and community composition and biodiversity (LeBauer and Treseder, 2008; Xia and Wan, 2008). Also new regional- or national-scale observational studies are highlighted that have systematically evaluated and synthesized data from hundreds to thousands of study sites (e.g., Simkin et al., 2016).

For atmospheric deposition of N or S impacts, EPA uses critical loads (CLs) to identify quantitative relations between chemical exposure (dose) and specific, quantitative changes in ecological properties or processes (EPA, 2020b, p. 92). The definition of a CL is, “a quantitative estimate of an exposure to one or more pollutants below which significant harmful effects on specified sensitive elements of the environment do not occur according to present knowledge” (EPA, 2020b, p. 95; Nilsonn and Grennfelt, 1988). EPA uses results from field experiments, but mostly from models including statistical regressions or simple mass balance equations (SMBEs), very simple dynamics (VSD), or more dynamic models to identify the critical load. The ISA Appendix presents estimates of critical loads of nitrogen for mycorrhizal fungi, lichens, and herbaceous plants, based on a 2011 U.S. Forest Service assessment (Pardo et al., 2011), with updates from more recent literature. Overall, the review indicates that the evidence for concluding that N deposition causes changes in productivity, carbon sequestration, and biodiversity of terrestrial ecosystems has been strengthened since 2008 by increased mechanistic understanding as well as by coherent evidence from new studies encompassing a wider range of organisms, regions, and ecosystems (EPA, 2020b, pp. 6-190–6-193).

The Integrated Synthesis section of the 2020 NO_x-SO_x-PM ISA (EPA, 2020b) includes a 10-page description of how uncertainty is considered for welfare effects, describing uncertainty caused by inherent variability and randomness and uncertainty associated with inadequacy of the science that can be reduced with further research. It was noted that the understanding and reporting of uncertainty differed across the many different disciplines with input into welfare effects; the ISA goes into detail for only some of the uncertainties associated with the relationships between NO_x, SO_x, and PM pollutants and ecological effects. These included source emissions measurements, atmospheric deposition estimates, empirical measurements of critical loads, models used to estimate CLs, and uncertainties in the aquatic acidification index that EPA developed for the 2008 NO_x-SO_x ISA (EPA, 2008c). For ecological impacts uncertainty was described for empirical modeled critical loads, including for terrestrial and aquatic acidification (biogeochemistry), biogeochemistry and plant diversity-linked modeling, and aquatic eutrophication modeling. Uncertainty in establishing the critical load for acidification from N and S deposition was acknowledged to come mostly from uncertainty in identifying base cation weathering rates and supply of acid-neutralizing capacity.

For N deposition effects on ecosystems there is uncertainty associated with measurements of wet and dry atmospheric deposition amounts and physical, chemical, and ecological variability across time and space for both terrestrial and freshwater environments.

EVOLUTION OF THE ISA PROCESS SINCE PUBLICATION OF THE PREAMBLE

The above discussion of evidence synthesis and integration includes examples from ISAs published more recently than the Preamble. In recent ISAs several new procedures are introduced that assist in screening, evaluating, and documenting study quality and relevance, adding confidence in the resulting causal determinations. EPA might commit to employing them consistently in future ISAs.

The first ISAs issued after the publication of the Preamble (EPA, 2015a) introduced detailed “study quality criteria tables” (Table A-1 in the health criteria ISAs for NO₂ [EPA, 2016] and SO₂ [EPA, 2017b]) that described EPA’s “scientific considerations for evaluating the strength of inference from studies on the health effects of” individual pollutants. Judgments about study quality were made without considering study results. The study quality criteria tables specified separate criteria for evaluating controlled human exposure, animal toxicity, and epidemiological studies. For each study type, the reviewers considered study design, study population/test model, pollutant, exposure assessment or assignment, outcome assessment/evaluation, potential co-pollutant confounding, other potential confounding factors, and statistical methodology. Similarly structured study quality tables have been included in more recent ISAs for particulate matter Table A-1 (EPA, 2019c), and ozone Table Annex 3-1 (EPA, 2020a), for health effects only. Given the broad range of welfare effects and the many different study types used to assess them, it may be difficult to develop similar quality criteria tables for evaluating welfare studies. However, similar scientific considerations might logically apply to evaluating the strength of causal inference from various types of welfare studies and could lead to future development of similar study quality criteria for welfare effects studies.

The most recent ISAs for particulate matter (EPA, 2019c) and ozone (EPA, 2020a) include appendices summarizing the “Development of the ISA” that provide more pollutant-specific detail than the Preamble and previous ISAs on the associated literature searches, study selection and quality evaluation, peer review, public comment, and quality assurance/quality control. The summary of the ISA development process in the 2020 Ozone ISA (EPA, 2020a) introduces several new tools that add consistency, transparency, and documentation in several areas of the ISA development process. One of these tools is a machine learning-based software tool called SWIFTActiveScreener (SWIFT-AS, where SWIFT is an acronym for Sciome Workbench for Interactive computer-Facilitated Text-mining) (Howard et al., 2020), which EPA employed to assist in initial literature screening (“Level 1” where the title and abstract are reviewed for indication of a quantifiable effect). Compared to traditional “hands-on” literature screening methods, SWIFT-AS identifies 95 percent of all potentially relevant references (performance comparable to human error rates), but saves time (critical in attempting to meet the 5-year National Ambient Air Quality Standards [NAAQS] review deadlines), and adds an element of reproducible objectivity to the initial literature screening process.

Another set of tools introduced in the 2020 Ozone ISA is called PECOS (Population, Exposure, Comparison, Outcome, and Study Design) (Morgan et al., 2018). The PECOS toolset helps provide a structured framework for refining the scope of the ISA and assists in decisions for including and excluding studies in the ISA by characterizing the parameters and identifying the evidence in the literature pertinent to the ISA. Discipline-specific PECOS tools were developed for experimental studies, epidemiologic studies, ecological studies. EPA notes (EPA, 2019a) that the use of PECOS

tools is consistent with prior reviews (NRC, 2009). PECOS was also extensively employed in the development of recent World Health Organization global air quality guidelines (WHO, 2021).

Studies considered for inclusion in ISAs are subject to several levels of screening. Level 1 screening is intentionally broadly inclusive and seeks to identify all potentially relevant literature, based on a review of study titles and abstracts. Level 2 screening includes full text evaluation by EPA Office of Research and Development subject matter experts. For the 2020 ozone ISA, level 2 screening decisions were based on scoping decisions defined for each topic area (e.g., outcome category) documented in the ISA (EPA, 2020a). Studies were included in the review by considering the extent to which the study was informative, pertinent, and policy-relevant to pollutant-effect relationships or the basis for such relationships. The emphasis was on concentration- or exposure-effect relationships in current populations or ecosystems at current ambient air concentrations, and particularly if unique data, such as a previously unreported effect or mode of action was reported. Decisions for including and excluding specific studies were documented in EPA’s Health and Environmental Research Online (HERO) database. HERO includes scientific references and associated data from the peer-reviewed literature. Information from HERO and data about concentrations, study design and results are reported in the appendices to the ISA. Additional detail on study quality evaluation was provided for the most “policy-relevant” studies (those associated with “causal” or “likely causal” determinations or supporting changes in prior causal categories). Narrative reviews for those studies were recorded in the Health Assessment Workspace Collaborative (HAWC) database³ and can be accessed from the HERO project page associated with the Ozone ISA (EPA, 2020a).⁴

Like the study quality criteria tables, first introduced in the 2016 NO₂—Health ISA (EPA, 2016), the SWIFT-AS, PECOS, and HAWC tools introduced in the 2020 Ozone ISA (EPA, 2020a) add consistency, transparency and documentation of literature review and inclusion to the ISA process and may be helpful to include in future ISAs.

___________________

³ See https://hawcproject.org (accessed January 22, 2022).

⁴ See https://hero.epa.gov/hero/index.cfm/project/page/project_id/2737 (accessed January 22, 2022).