Advancing the Framework for Assessing Causality of Health and Welfare Effects to Inform National Ambient Air Quality Standard Reviews (2022)
Chapter: 10 Conclusions and Recommendations
10
Conclusions and Recommendations
As prescribed by the Clean Air Act (CAA; 42 U.S.C. § 7408) and its Amendments, the U.S. Environmental Protection Agency (EPA) sets National Ambient Air Quality Standards (NAAQS) for criteria air pollutants that are reviewed periodically. The Integrated Science Assessment (ISA) is an important step in this review process in that it focuses on the science. Central to the ISA are causal determinations that link the presence of the pollutant in the atmosphere that is under review with health and welfare outcomes identified in the scientific literature. The causal determination framework—described in the Preamble to the ISAs (EPA, 2015a) guides the conduct of the ISA and has evolved over the multiple reviews of each of the criteria pollutants. The framework has been subject to criticism, motivating EPA to request the study resulting in this report. The committee’s major conclusions and recommendations are summarized below. A theme in those conclusions is the importance in transparency—both in the reporting of individual studies used to inform the causal determinations, and in the causal determination process itself.
Development of causal determinations, as described in the Preamble (EPA, 2015a), involves synthesizing existing evidence to characterize the weight of evidence regarding causality. This weight of evidence approach is then used to determine which of five causal categories is assigned to describe how strongly the scientific literature links the presence of the pollutant to an outcome. There have been criticisms of both how the weight of evidence approach is used, and the use of the five categories (see Chapter 6). With consideration given to these critiques, the study committee examined the effectiveness of the current EPA causal determination framework for developing ISAs to support the NAAQS review process. The committee also considered how other frameworks are used to conduct similar reviews and how causal determinations are made and supported using them. Alternative and potentially new methods to assess causality were also considered. Two broad conclusions were drawn:
Conclusion: A single study will rarely definitively and comprehensively address issues associated with the determination of causality that are examined in ISAs. A weight of evidence approach—combining assessment of study quality with expert judgment—allows EPA to draw conclusions that integrate scientific findings across multiple study designs and disciplines, as
required by the CAA. Increased transparency in how evidence is integrated would improve confidence and understanding of the ISA causal determinations and other conclusions.
Drawing causal determinations for the effects of air pollution exposures, where both complex exposures and complex biological mechanisms are involved, is challenging. As described in Chapter 4 and elsewhere in this report, a variety of study designs are used to study health and welfare effects, each with its own strengths and weaknesses. A single study can rarely address sufficiently all variables. For example, randomized experiments are strong in terms of their ability to isolate causal effects, but they generally are not conducted in the full populations of relevance for an ISA. On the other hand, large-scale epidemiologic studies will provide data on broad populations and exposures, but will require strong non-experimental methods to generate robust conclusions. Each study needs to be assessed in terms of its design and analysis strengths and limitations, and how its design and results fit with other knowledge and data. A weight of evidence approach, which combines assessment of the scientific literature with expert judgment to weigh that complex literature, is a scientifically defensible approach for the ISA causal determination framework.
Conclusion: The five causal categories in the hierarchy to classify weight of evidence for causation as described in the Preamble are able to characterize the strength of evidence regarding causality in the ISA assessments.
The study Statement of Task (see Box 1.1) directs the present study committee to consider the number of causal categories used by EPA in the NAAQS process. Evidence reviewed by the committee suggests that the granularity of the current causal categories (i.e., “causal relationship,” “likely to be a causal relationship,” “suggestive of, but not sufficient to infer, a causal relationship,” “inadequate to infer the presence or absence of a causal relationship,” and “not likely a causal relationship”) is scientifically meaningful and useful in the context of the NAAQS reviews. The five-level framework qualitatively expresses the degree of uncertainty in the causal determinations, and the associated rubric summarizes key strengths and limitations of the body of evidence. The distinctions help EPA determine which health or welfare endpoints should be carried into the risk and exposure assessment stage of the NAAQS review process (e.g., EPA, 2010, 2016). The five causal categories used by EPA cover the range of foreseeable scientific scenarios considered in NAAQS reviews and are consistent broadly with those of other regulatory and health and environmental guidance groups (see Chapter 7). Furthermore, the causal categories are scientifically defensible given the intended protective nature of the NAAQS and EPA’s legal responsibility to act in a precautionary way “to protect and enhance the quality of the Nation’s air resources so as to promote the public health and welfare and the productive capacity of its population” (42 U.S.C. § 7401(b)(1)).
Although some critics have called into question EPA’s “likely causal” category (see Chapter 6), this category is meaningfully distinct from the other categories and is useful during the NAAQS process because it indicates a likely causal effect, but that the remaining uncertainty after reviewing the evidence inhibits making the determination that a causal association has been demonstrated. Other causal determination frameworks include categories that are analogous to the “likely causal” category, for example, the “probably carcinogenic to humans” used by the International Agency for Research on Cancer (IARC, 2006), which also uses a weight of evidence approach, and uses this category to characterize an association that is not fully established by the studies reviewed. The causal classification schema of some of the other frameworks have four categories, including a “likely causal” category, but they omit a category analogous to EPA’s “not likely to be causal” (e.g., IARC [2006] and the 2013 Canadian Smog Assessment [Health Canada, 2013b]). However, given that such a conclusion could be accurate and scientifically meaningful for a given NAAQS
review process, that category is useful for ISAs, and informs future ISAs. More recent Canadian air quality assessments for sulphur dioxide and for nitrogen dioxide, both completed in 2016, consider five causal categories, identical to those employed by EPA (Health Canada, 2016a,b).
IMPROVING THE SINGLE CAUSAL DETERMINATION FRAMEWORK FOR HEALTH AND WELFARE EFFECTS
The Statement of Task (see Box 1.1) specifically called on the committee to address whether a single framework and practices related to it for assessing causality may be applied to both health and welfare effects. The committee considered causal assessments in past ISAs and how the framework might be applied to develop causal determinations to support the NAAQS review process. Considerations for improving the effectiveness of a common framework for use in both health and welfare causal determinations are discussed in Chapter 9, but the following conclusions are provided here:
Conclusion: The Preamble’s causal determination framework can adequately guide causal determinations for both health and welfare as long as comprehensive and well-defined scientific questions and pollutant exposure-outcome relationships are identified and addressed.
A primary purpose of the ISAs is to compile, review, synthesize, and clearly convey the scientific information on the causality of health and welfare effects of air pollution that are relevant to NAAQS decision making under the CAA. Core scientific principles of causal assessment cut across health and welfare effects (see Chapter 9). Questions of exposure assessment, study design and quality, and transparency and replicability are equally relevant to assessing causality of health and welfare effects, even given that the types of studies that inform health or welfare assessments may differ substantially. Using the same framework for both health and welfare effects allows a uniform approach to assessing these two areas of air pollution impacts; benefits of this include enhanced ability to examine and integrate the linkages between health effects and welfare effects.
However, there is opportunity to improve the ISA causal determination framework so that the resulting ISAs more effectively convey the state of scientific understanding. This includes providing guidelines in the framework to ensure that causal determinations are adequately supported and explained and that the significance of relevant exposure patterns and welfare endpoints evaluated is made clear. The framework may need to be modified to be certain that ISAs are able to address the relationships between exposure metrics that are scientifically relevant and well defined to support causal determinations for public health and welfare endpoints. The framework might be modified to include assessment of the relevance of fundamental scientific questions identified during the initial stages of the NAAQS review process, particularly as literature is identified and reviewed during the ISA and causal determination. The adequacy of those scientific questions toward supporting the causal determinations needs to be conveyed and documented as part of the process. For example, the 2019 Ozone Integrated Review Plan (IRP) (EPA, 2019a) includes detailed discussions of the level, form, indicator, and averaging time of the existing standard in relation to health effects, but there are no parallel questions raised about the appropriateness of those elements of the standard in relation to welfare effects.
As another example, causal determinations have focused on current effects of criteria pollutants or on conditions that have occurred in the recent past. For many welfare effects, such as those resulting from sulfur and nitrogen deposition, and for some health effects, such as those due to lead, long-term cumulative impacts are important (see Chapter 9). Looking forward, climate change can alter atmospheric chemical transformations, transport, and ecosystem processing of air pollutants,
impacting many welfare effects. The impacts of climate change on health effects are becoming more apparent. The sensitivity of people and ecosystems to atmospheric contaminants may also change. In this regard, it is important to consider future environmental conditions when addressing the impacts of criteria pollutants on health and welfare, and the framework might include guidance on how additional scientific knowledge gained during the ISA process regarding the adequacy and thoroughness of the scientific questions identified in the IRP can be incorporated into the causal determinations. In the example related to climate change, information from paleoecological studies of ecosystem responses to climate change and the use of simulation to project past and future exposure and effects of air pollutants might be incorporated into causal determinations.
Conclusion: Heterogeneity in the response of individuals and populations exposed to air pollution complicates causal assessments. The current framework separates description of vulnerable groups and sensitive ecosystems or species from causal determinations, potentially obscuring understanding of causal relationships for the more exposure-sensitive groups of subjects when the causal category determinations are presented at a broader level.
Recommendation 1: Include guidelines in the framework regarding how heterogeneity in exposure responses is considered to ensure causal determinations fully account for evidence of effects in sensitive groups of humans, other species, and ecosystems. To the degree practical, the framework should provide guidance on how evidence should be examined for key sensitive groups and then integrated across endpoints or subgroups in establishing causal determinations.
There is heterogeneity in both the health and welfare responses of individuals, populations, species, and ecosystems being exposed to pollutants. Heightened response in humans can be due to age, comorbidities, or other environmental, socioeconomic, behavioral, epigenetic, or genetic factors. Similarly, species and subspecies responses to exposures can vary, and susceptibility and resilience can be influenced by a suite of environmental factors (e.g., buffering capacity, heat and water stress, and the presence of other stressors; see Chapter 9). Considering, or highlighting, only overall average population or broad ecosystem effects can obscure causal relationships that exist for more sensitive subgroups, subspecies, communities, or ecosystems. The aim of advancing environmental justice requires enhanced consideration of heterogeneity in health responses linked to socioeconomic status, race and ethnicity, and community- and individual-level social determinants of health. There are threatened or endangered species and critical habitats or ecosystems, including those at the global scale, that warrant heightened attention (see Chapters 4 and 9). Furthermore, studies may differ in their treatment of subpopulations, or whether critical subpopulations are represented at all. For example, controlled human exposure studies generally exclude subjects with serious health conditions, and controlled exposure and panel studies may not be able to recruit sufficiently diverse participants to allow generalization to groups at heightened risk (see Chapter 4). Weight of evidence assessments need to account for selection bias in individual studies and across different lines of evidence. See Box 10.1 for a description of potential research in this area.
STUDY QUALITY EVALUATION
Individual studies are the building blocks for causal assessment. To optimize how evidence used in a causal assessment is weighed, integrated, and synthesized, a causal determination framework needs to include information on how to assess and document the relevance and quality of individual studies that are ultimately selected for inclusion in the causal assessment. Included in the assessment of relevance and quality is consideration of different study designs, including how studies account for the variety of confounders that may be present.
Conclusion: The causal determination framework described in the Preamble provides general guidance for individual study quality evaluation, but minimal detail regarding determination of individual study relevance, study inclusion or exclusion, or influence on weight of evidence causal determinations. Specific criteria for assessing the quality and relevance of individual studies to inform a given causal determination are described in recent IRPs, but the formal framework provides little direction for developing or forming those criteria for particular causal determinations.
Recommendation 2: Include in the causal determination framework used for developing Integrated Science Assessments a set of foundational study design attributes and analysis approaches to be considered when selecting and evaluating studies used for causal determinations. Include discussion of the attributes examined and how the resulting examination (e.g., the specific study characteristics of interest) influences the consideration of individual studies in the weight of evidence approach.
Scientifically meaningful implementation of a weight of evidence approach requires a clear process for selecting studies for inclusion and for evaluating their quality and relevance for the causal question being examined. While the details of how studies will be evaluated and weighed will depend on the particular pollutant and causal questions, there are foundational aspects of study quality as they relate to questions of causality, as discussed in Chapter 3 and Appendix C. Clarifying what aspects of a study will be considered in assessing its relevance and quality—in a framework for causal determination—can help increase the transparency and replicability of the study selection, and evaluation process.
The 2020 Ozone ISA (EPA, 2020a) provides detailed quality review summaries for the 140 health effects studies considered most “policy-relevant” (those studies supporting “causal” or “likely causal” determinations or resulting in a change in causal category from the previous ISA), along with associated “study evaluation requirements.” The summaries are recorded in a Health Assessment Workplace Collaborative (HAWC) database1 and can be accessed from the Health and Environmental Research Online (HERO) database project page associated with the ozone ISA.2 These summaries are useful for understanding how the studies were assessed in making causal determinations. However, the process has not been systematized, and the exact bases for studies’ inclusion, exclusion, or selection as particularly relevant to causal determinations are unclear.
Tools used in the 2020 Ozone ISA (e.g., HAWC, Population, Exposure, Comparison, Outcome, and Study Design [PECOS], HERO) provide helpful explanation and documentation of criteria for assessing study relevance and quality. While it is not appropriate to use the outputs of such tools as decisive benchmarks for inclusion in causal determination, their continued use and refinement would improve clarity regarding the study selection and evaluation process. The key aspects of study quality and relevance that are assessed in the weight of evidence approach for the causal question under consideration may then be documented. The exact criteria may be pollutant, study type, or endpoint specific, so any individual tool may not be applicable for every causal determination, and specific tools will evolve and new ones may be developed. Therefore, it may be inappropriate for the framework to prespecify use of any particular tool, although the framework could include a set of core scientific principles regarding study inclusion and quality to increase transparency and replicability.
The Preamble’s causal determination framework would benefit from formalization of criteria to assess study validity, and the individualized use of tools for each ISA (such as PECOS, study quality criteria tables, and narrative study quality reviews) to implement those criteria. Such criteria have been used by the U.S. Department of Health and Human Services’ National Toxicology Program’s Office of Health Assessment and Translation (NTP, 2019); EPA’s Integrated Risk Information System (IRIS; EPA, 2020b); the Toxicological Data Reliability Assessment tool (ToxRTooL; Schneider et al., 2009); the Strengthening and Reporting of Observational Studies in Epidemiology (STROBE) guidelines (von Elm et al., 2007); and the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines (Percie du Sert et al., 2020). Similarly, PECOS tools were employed in recent international air quality health assessments by the Health Effects Institute (2019) and World Health Organization (WHO, 2021). Chapter 7 provides some details on criteria and tools used by other causal determination frameworks, and EPA might further explore approaches used by some of those other frameworks for study selection and quality and relevance. As one example, some other causal determination frameworks evaluated by this committee (see Chapter 7; e.g., the WHO global air quality guidelines) publish in advance the specific protocol that will be followed during the systematic review; this is known as “preregistration” and can be done using international registries such as PROSPERO. Preregistered protocols should include the specific criteria for literature searches, the types of study to be included or excluded, the populations under study, exposures, outcomes, risk of bias assessment tools (if used), strategies for data synthesis, etc. Elements of this approach could be adopted to increase the transparency of the ISAs.
Scientific principles and an articulated framework for assessing study quality and applicability for making causal determinations, such as those laid out in Chapters 3 and Appendix C, rather than prescriptive study eligibility criteria are suggested in this report because it is important that the causal assessment framework allow scientific flexibility. A recent review of EPA’s staff handbook for developing IRIS assessments recommended against using “the results of study evaluation as criteria for … systematic review” (Recommendation 4.1; NASEM, 2022). No single study selection
___________________
1 See https://hawcproject.org/about (accessed February 9, 2022).
2 See https://hero.epa.gov/hero/index.cfm/project/page/project_id/2737 (accessed February 9, 2022).
or evaluation tool should prescriptively include or exclude eligibility for inclusion of a study in the ISA. However, articulating the study attributes of relevance for assessing causality can assist in the weight of evidence approach.
Recommendation 3: Provide explicit guidance in the causal determination framework for (a) assessing the approaches used in individual studies to account for important and potentially biasing confounders, and (b) how the strength of those approaches might influence weight of evidence considerations in causal determinations.
The current framework for causal determinations as described in the Preamble (EPA, 2015a) recognizes that copollutants may be confounding factors when assessing the potential effects of a criteria pollutant, but the framework is not explicit about other types of confounding, such as confounding by weather effects, other environmental factors, or socioeconomic or demographic differences within populations. Guidance in the framework regarding a few key aspects of confounding would help to improve the scientific conclusions drawn in the weight of evidence approach. In particular, when evaluating individual studies, the weight of evidence approach could take into account
- how well a study articulates concerns about confounding and what the relevant confounding factors are (e.g., whether the study includes a conceptual framework for potential confounders of the research question of interest);
- whether the relevant confounders are observed and adjusted for in the study design and analysis using scientifically meaningful statistical methods (e.g., whether key confounders defined in the scientific literature are addressed and how well the study design and analysis incorporates observed confounders); and
- whether analyses of the robustness of study results to an unobserved confounder have been conducted, and how such an unobserved confounder might change study conclusions.
In this way the causal determination framework could provide a structure for assessing individual study quality in terms of this ability to control for key confounding factors.
Conclusion: EPA recognizes the importance of replicability of individual studies when making causal determinations. However, the Preamble’s causal determination framework does not provide explicit guidance regarding how the potential reproducibility and replicability of individual studies should affect the influence of those studies on causal assessments.
Recommendation 4: Develop guidance for the causal determination framework for assessing individual study documentation of data, methods, and assumptions, and for how the use of that assessment informs the influence of the individual study in the weight of evidence approach.
Reproducibility and replicability of scientific results are important when gauging the rigor of a scientific study (NASEM, 2019). As described in Chapter 4, replicability indicates that consistent study results are observed across studies conducted with different data. Reproducibility—highly related to transparency—refers to the ability to obtain the same results given the same data. The ability to reproduce results across studies using different data is recognized in the Preamble as constituting “one of the strongest arguments for causality” (EPA, 2015a, p. 20),3 and in some ways this concept is the foundation of the weight of evidence approach; concordance of findings across
___________________
3 Note that the Preamble (EPA, 2015a) defines “reproducibility” as NASEM (2019) defines “replicability.”
the scientific literature is the basis of the Preamble’s causal assessments. However, the framework does not describe how to assess transparency and potential reproducibility of individual studies in the weight of evidence approach. For example, studies in which both study design and data collection are explicitly described are, by their nature, more reproducible than studies that are unclear about their designs, data collection, and analysis methodologies. Transparency and the ability to reproduce study results can help increase confidence in study results. Attention to transparency and reproducibility is particularly important as the complexity of study designs increases; for example, analyses to account for time-varying, large-scale data are complex. Transparency is greater in a study conducted within an open science framework with fully available data and statistical code, however, some data sharing and privacy considerations may not always make this feasible. In such cases, at the very least, the statistical software and code used to analyze the data could be documented and shared. Similarly, results that are replicable and seen across variations in study designs or specific analysis choices can be viewed as more robust and as strong evidence for a causal relationship. EPA should investigate how study transparency, reproducibility, and replicability should influence study quality and relevance assessments used in the weight of evidence approach. The causal determination framework should then be modified to articulate the role of study transparency, reproducibility, and replicability should be assessed when considering individual study quality and relevance in the weight of evidence approach.
INTEGRATING EVIDENCE
Recommendation 5: Monitor (a) research in the scientific literature on evidence integration and (b) the evolution of other frameworks used to assess causality (e.g., IRIS and OHAT) to determine if any emerging approaches or characteristics of their research synthesis and evidence integration protocols might be adapted to improve Integrated Science Assessment causal determinations for health and welfare.
Across public health, medicine, and environmental sciences, there is a spectrum of approaches for integrating evidence to develop an overall causal determination. At the individual study level, various risk of bias tools may be used to provide ratings of dimensions of study quality and relevance (which may be numeric); these include tools developed within recent ISAs and similar approaches (see Chapter 7). However, development and application of such tools depends on expert judgment; what inputs are required and how should they be weighted, and then what are the specific inputs for a given study. Even when inputs are apparently clear-cut (e.g., binary yes-or-no questions), expert judgment may nevertheless be required to determine how a study should be coded in terms of those inputs. Within a line of evidence, certain types of studies may be combined using formal methods, such as meta-analysis (for some types of epidemiological studies), or a modified Grading of Recommendations Assessment, Development and Evaluation (Schünemann et al., 2013) approach as used by WHO (2021). However, choosing studies on which to apply such formal tools, and what to do in the face of heterogeneity in results, ultimately depends on expert evaluation. These tools may be particularly challenging to apply in complex settings, such as environmental epidemiology, with a wide range of research questions, study designs, timescales, and potential confounders. Where there are mixes of study types within a line of evidence, or when combining evaluations across lines of evidence, no single formal tool is available that does not ultimately depend on expert evaluation for its inputs (e.g., Delphi methods, or the Analytic Hierarchy Process). No tool is recommended for use in any of the causal determination frameworks examined by the committee.
The framework for causal determinations outlined in the Preamble offers a reasonable middle ground approach for evidence integration given the complex questions under study in making causal determinations in the ISA process. It is broadly consistent with the frameworks and
approaches used by many other groups studying similarly complex causal questions (see Chapter 7), no doubt partly because the authors of such frameworks consult other examples, including the ISAs. It will always be difficult to place explicit weights on individual studies and then on the combination of different types of studies, somehow considering all aspects of the study quality and relevance. Expert judgment is necessary at all points of the process, often in ways that defy formal evaluation. Determining the correct weighting structure to use, let alone the actual weights assigned, is necessarily challenging in complex settings such as that studied in ISAs; and there is no evidence to show that application of more formal methods does or could provide either a more reliable or better explainable result than the consensus approach currently in use in all such frameworks examined by the committee.
Given the complex setting, EPA should continue to monitor developments in research synthesis and evidence integration that might be applied to a weight of evidence approach. Further research could identify whether there are approaches that could be borrowed from other disciplines or frameworks (see Chapter 7) that provide structures for combining different sources of evidence from a variety of disciplines. Recent advances in decision making, such as multi-attribute decision analysis, may eventually be useful for the ISA process, but will require more research and assessment for application in making causal determinations to determine whether they can provide better support for expert consensus evaluations (as determined, for example, by Clean Air Scientific Advisory Committee [CASAC] reviews) or provide better documentation of the overall process. Similarly, formal meta-analysis or risk of bias tools may be meaningful for certain subsets of the evidence (as used in some of the frameworks examined by CASAC), which could then be integrated in with the other streams of evidence. Finally, EPA should monitor related fields such as risk assessment, systematic review, and decision analysis to identify any advances in practical application of expert judgment acquisition and documentation. Further transitioning toward formal risk of bias and quantitative rating approaches needs to be carefully evaluated with substantial vetting and research on consequences, and particular care should be taken to distinguish whether welfare endpoints should continue be treated in similar fashion to health endpoints if such an approach were taken.
EXPERTISE
Conclusion: Given the broad range of topics covered when determining causality for criteria pollutants, access to a broad range of expertise within and outside EPA—including expertise in emerging areas such as causal modeling and inference—is needed throughout the causal determination process to ensure incorporation of the latest scientific knowledge.
The pace of evolution of the science and methods to study questions of study relevance when making causal determinations is rapid, as are the many scientific facets of how pollutants impact public health and welfare. Therefore, it is crucial that EPA stay abreast of the science. However, a major impediment to staying abreast of the state of the art in science is the existence of silos of scientific expertise and organizational structures. These impact the transfer and use of knowledge across groups. The silos of science are a problem endemic to science, and EPA is not immune. One way to help overcome this during the conduct of an ISA is to ensure that the expertise needed in all relevant fields at different phases of the ISA process as laid out in the Preamble are identified and included in the conduct of the causal assessment.
Recommendation 6: Articulate in the causal determination framework a clear process for identifying and incorporating the necessary expertise—including expertise in relevant emerging areas—for each step of the causal determination (e.g., development, individual study selection and assessment, through final review).
Scientifically well-founded causal determinations made via a weight of evidence process requires expertise in a wide range of scientific disciplines such as exposure science, epidemiology, social sciences, clinical trials, mechanistic studies, ecology, agricultural sciences, and [bio]statistics including causal modeling (note that this list is not exhaustive and may differ for each ISA). The need for this range is evidenced by the breadth of topical areas covered in a typical ISA and the large literature body utilized when making causal determinations (see Chapters 4, 5, and 8). While EPA has procedures to identify experts both to participate in drafting the ISAs that support the causal determinations and in forming the EPA-appointed bodies that conduct the reviews, the Preamble should systematize a process for assuring that all the scientific disciplines necessary to develop and thoroughly review causal determinations as described in the Preamble’s framework are identified. This is particularly important given the increasingly voluminous and complex ways data are collected, assembled, and analyzed. When identifying experts, EPA should ensure that a range of perspectives in each of those scientific disciplines are objectively incorporated into the ISA process.
The convening of many scientific bodies to conduct other reviews is often driven by guidelines for identifying and selecting the reviewers. For example, as specified in the IARC Monograph Preamble (IARC, 2006), IARC selects members for each monograph working group to ensure each group “is interdisciplinary and comprises subgroups of experts in the fields of (a) exposure characterization, (b) cancer in humans, (c) cancer in experimental animals, and (d) mechanistic evidence” and encourages “public nominations through its ‘Call for Experts.’” (Given that causal determinations being made as part of NAAQS reviews typically includes more endpoints for both public health as well as welfare, a broader range of expertise will be required.) Likewise, the National Academies of Sciences, Engineering, and Medicine have an established committee appointment process with formal steps to identify and vet needed expertise, range of perspectives, and experts for any of its consensus studies, workshops, or standing committees.4 Expertise is identified and experts chosen through a process that allows input from multiple levels of the institution and at multiple points during the selection process. A public vetting of the nominated committee occurs before committee selection is finalized.
A hallmark of the current causal determination framework is the extensive, iterative review of the underlying documents by a broad range of EPA-appointed and public reviewers and commenters. Providing systematized guidelines in the Preamble for identifying the scientific disciplines and perspectives needed for rigorous objective consideration of all the science will increase confidence in the resulting causal determinations.
CONCLUDING THOUGHTS
The Statement of Task (see Box 1.1) for this study did not ask the committee to provide EPA a new causal determination framework, but rather to identify approaches that could be incorporated by EPA into a future iteration of the causal determination framework. The ISA causal determination framework is not a procedure that can be tested objectively or evaluated against the ground truth. Through a consensus process, the committee assessed whether it is possible, using the current ISA framework, to arrive at scientifically sound causal determinations given existing evidence (recognizing that a different causal conclusion might be arrived at as new evidence is generated). The committee concludes that the fundamental structure (a weight of evidence approach based on review of the scientific literature, extensive internal and external review, and five causal categories) of EPA’s causal determination framework as described in the Preamble is able to support causal determinations made as part of the NAAQs review process assuming comprehensive and well-defined scientific questions and pollutant exposure-outcome relationships are identified (including
___________________
4 See https://www8.nationalacademies.org/pa/information.aspx?key=Committee_Appointment (accessed May 8, 2022).
for key sensitive subgroups), and assuming the expertise with a range of perspectives needed are engaged in the conduct of the ISA (including in emerging areas relevant to the ISA). A single causal determination framework for both health and welfare effects is reasonable given identification and attention to considerations related to both health and welfare endpoints (e.g., different types of study designs used and the timescales and outcome assessment of relevance). The number and descriptions of causal categories provided in the Preamble’s framework can express the degree of uncertainty in the causal determinations. The distinctions between categories allow EPA to then consider which health or welfare endpoints should inform the risk and exposure assessment stage of the NAAQS process. However, the framework needs to provide more guidance with respect to addressing uncertainties related to heterogeneities in exposure response of individuals, populations, and sensitive ecosystems directly into the determination of causality. It may need to be modified so that ISAs are able to address the relationships between exposure metrics that are scientifically relevant and well defined to support causal determinations for public health and welfare endpoints and to incorporate sound new methodologies for evidence integration that may be described in the scientific literature or adopted for use in other causality determination frameworks that would improve the weight of evidence approach.
The United States benefits from a world-class air pollutant observation network that supports research and understanding of the public health and welfare impacts of elevated pollutant concentrations. The framework applied by EPA for making causal determinations guides the integration of the knowledge gained from that network, and from other data resources and a broad array of study designs, to inform causal determinations. This report provides insights regarding strengths and weaknesses of the current causal determination framework, and indicates where modifications could increase transparency of the ISA process itself. It suggests where expertise might be expanded in the ISA process, and describes ways to enhance the weight of evidence approach applied in causal determinations.
This page intentionally left blank.
