Appendix E

A Brief History of Ecological Risk Assessment

The field and practice of ecological risk assessment has evolved substantially over more than a quarter century. The field of risk assessment, whose origin in the United States is summarized in the 1983 National Research Council (NRC) report Risk Assessment in the Federal Government: Managing the Process (widely known as “the Red Book”), preceded the emergence of ecological risk assessment. In the beginning, the main stressors of interest were chemicals and the endpoints of interest were cancer and human health; ecological effects were not a part of this initial formulation. By the late 1980s, there was growing interest in ecological processes and effects, prompting US Environmental Protection Agency (EPA) to begin preliminary work on guidelines for risk assessment focused on ecological effects of stressors. In the early 1990s, EPA generated a framework and guidance documents for the conduct of ecological risk assessment (EPA, 1992, 1998), and similar guidance documents were developed in Europe, Canada, and Australia.

In 2006, the Ecological Processes and Effect Committee of the EPA Science Advisory Board held a workshop on the current and future practice of ecological risk assessment that led to four important publications: Suter (2008) summarized the history of the development of ecological risk assessment from the mid-1980s to the mid-2000s; Barnthouse (2008) outlined the strengths of ecological risk assessment; Kapustka (2008) detailed some of its limitations as they stood in the mid-2000s; and Dale et al. (2008) provided a list of conclusions and recommendations for improving ecological risk assessment and its use in the decision-making process. The workshop and the subsequent papers brought to the forefront key aspects of ecological risk assessment and ways to improve it. For example, Dale et al. highlighted the critical importance of communication with decision makers and stakeholders during the development of endpoints and management questions, and recommended that a peer review be conducted at the problem-formulation stage. Having appropriate endpoints and management questions is essential to the ability to accurately describe cause-effect pathways and inform decision making. The workshop also underscored the importance of analyzing and reducing uncertainty to increase the predictive power of the risk assessment. Dale et al. suggested that risk assessment and monitoring programs should be better integrated, and recommended post-cleanup assessments to facilitate this. The workshop also called for methods to quantify the weight-of-evidence process, and recognized that ecological risk assessments should include the effects of chemical and non-chemical stressors at various organismal and ecological levels of organization and spatial scales. Finally, the workshop identified the need to develop methods to estimate cumulative risk assessments, together with techniques to deal with the reality that a number of stressors exist in the environment, not just the one of current regulatory interest. Several of these recommendations were echoed in the 2009 NRC report Science and Decisions: Advancing Risk Assessment. The report, a comprehensive review of EPA’s human and ecological assessment process, remains an important milestone in assessing contemporary risk assessment and structured decision making within EPA.

Recently, Greenberg et al. (2015) reflected on contemporary practice in risk assessment since the NRC’s Red Book report. Concluding that many of the Red Book’s recommendations still hold, the authors noted that the view of risk assessment presented in the report has proved applicable to a much broader variety of circumstances than it was explicitly intended to address. For example, although the Red Book discussed risk assessment in the context of chemical exposures and a narrow set of effects, the process has been used for engineering, ecological effects,

and other fields, and the oil, rail, chemical, aerospace, and medical fields have adopted risk assessment as a standard practice. However, the application of risk assessment in the context of ecology has been somewhat more limited, perhaps reflecting a lack of understanding by risk assessors that ecological systems are nonlinear, complex, uncertain, and dynamic, and that outcomes are determined by multiple sources of stress. A symptom of this limited vision, for example, may be the late recognition of the importance of climate change (Landis et al., 2013) in evaluating risk to large-scale systems.

The following sections discuss the evolution of key aspects of ecological risk assessment, as well as specific applications, that may help to inform risk assessment approaches for gene-drive modified organisms.

CUMULATIVE RISK ASSESSMENT

Chapter 7 of the 2009 NRC report Science and Decisions: Advancing Risk Assessment, titled Implementing Cumulative Risk Assessment, is especially pertinent to the risk assessment of gene-drive modified organisms, because it defines cumulative risk assessment and elucidates the importance of expanding risk assessment beyond a narrow focus on a specific stressor.

Two methods for performing cumulative risk assessment have been described. An approach known as stressor-based cumulative risk assessment (Menzie et al., 2007) focuses on integrating multiple stressors, management options, and endpoints into a conceptual model that is used as the basis of risk assessment. The method uses the conceptual model to evaluate the likely stressors, their sources, and combinations of interactions that may occur. In the four-step assessment process outlined by Menzie et al., steps 3 and 4 focus on the range of management options. The NRC report Science and Decisions: Advancing Risk Assessment (2009) proposed modifying this approach to reduce the number of interactions to be considered, given that many of the original considerations included in the method would not be amenable to management and therefore are less pertinent to the risk assessment process.

The other method of cumulative risk assessment is the relative risk model (RRM) proposed by Landis and Wiegers (1997). This approach uses a ranking system to combine the interactions between multiple sources, stressors, habitats, and effects to estimate impacts to ecological structures. Wiegers et al. (1998) applied this approach to the Exxon Valdez oil spill and its effects on Port Valdez, Alaska. Since then, assessments using the RRM have been completed for a variety of stressors and combinations of stressors including contaminants, disease, environmental parameters, non-indigenous species, and the evaluation of landscapes (Walker et al., 2001; Moraes et al., 2002; Hayes and Landis, 2004; Colnar and Landis, 2007; Bartolo et al., 2012; Ayre et al., 2014; Hines and Landis, 2014; Allen et al., 2015; Heenkenda and Bartolo, 2015; Kanwar et al., 2015). Ayre and Landis (2012) also demonstrated how the RRM could be applied to management options. Since the early 2000s, Monte Carlo sampling has been used to describe uncertainty and to identify those variables that have the biggest impact on risk (Landis and Wiegers, 2005).

APPLYING ECOLOGICAL RISK ASSESSMENT TO INVASIVE SPECIES

In many ways, the release of gene-drive modified organisms is similar to the movement of invasive species. The early application of ecological risk assessment for the evaluation of invasive species was described in Andersen et al. (2004a,b), which stemmed from a workshop convened to bridge the gap between risk assessment as described in EPA’s 1998 guidance and the evaluation of invasive species. Several points from Anderson et al. (2004b) are especially relevant to gene drive research:

• There is a need for a conceptual basis for identifying assessment endpoints, determining the appropriate spatial and temporal scales, and describing the complexity of the resulting model.

• There is a need for guidance regarding the use of analytic tools; Bayesian or resampling (Monte Carlo) approaches are preferred.
• Multi-scale, spatially-explicit support systems, such as geographic information system mapping data tied to landscape population models, enhance credibility and make clear the trade-offs and the costs of inaction, thus supporting better decision making.
• The basic framework for an invasive species risk assessment (outlined in Figure 6-1 of this report) follows a source-exposure-habitat-effects−impact structure. For invasive species, the source is the native range of the species, exposure is the transport, habitat is the port of entry, and effects would describe the demography of the invasive species. The impact refers to the effects that are due to the presence of an invasive population.

Landis (2004) expanded the original framework proposed by Andersen et al. (2004b) into a generic conceptual model for invasive species following the basic formula previously used for the relative risk model (Landis and Wiegers, 1997). Modifications address the specific factors important to dealing with invasive species, and propose a basic computational framework for calculation using a Monte Carlo approach. Colnar and Landis (2007) used this framework to detail the risk posed by the invasion of the European Green Crab in the Northern Puget Sound at Cherry Point, Washington. The conceptual model was spatially specific and included multiple stressors and multiple endpoints. For some endpoints, the European Green Crab provided a negative risk (i.e., a benefit), for example, because it represented an additional food resource for native animal populations. However, the invading crab was determined to be detrimental in regard to other endpoints, such as those related to effects on native crab species and habitat.

Herring et al. (2015) applied the same basic structure but used Bayesian networks to assess risks posed by invasive species in the Padilla Bay National Estuarine Research Reserve in Anacortes, Washington. Puget Sound is already colonized by a large number of invasives and serves as a source of input to Padilla Bay. Evaluating potential mitigation strategies, the case study found that the treatment of ballast water at two nearby refineries would not substantially reduce the risk due to invasive species.

RECENT DEVELOPMENTS IN ECOLOGICAL RISK ASSESSMENT

A three-day workshop held at the Sydney Institute of Marine Sciences in Sydney, Australia in September 2014 provided a forum for examining the state of ecological risk assessment, identifying limitations of current practice, and proposing criteria for future assessments, with a focus on evaluating risk in the context of multiple stressors at large spatial scales as integrated into an adaptive management scheme. Van den Brink et al. (2016) presented findings and recommendations from the workshop, which are summarized here.

A major limitation identified by workshop attendees was that ecological risk assessments have been focused on single stressors affecting only a few receptors over relatively small spatial scales. However, many systems are affected by numerous abiotic and biotic factors, including disruption of the landscape by development, the introduction of non-native species, and the use of multiple agricultural chemicals. If only the stressor of primary interest is included in the risk assessment, the assessment will overlook interactions with other stressors and the risk will be presented out of context. In addition, ecological risk assessments often have not appropriately accounted for the fact that the intensity of the stressor will vary by location and over time. Indirect effects may also play a critical role and in some cases can be more influential that direct effect on the endpoints. Specific limitations of many ecological risk assessments include:

• Inherent limitations stemming from a lack of knowledge, as well as contrived limitations stemming from outdated guidance and regulations on conducting ecological risk assessments.

• A lack of useful data needed to answer questions specific to the risk assessment question, especially a lack of site-specific data.
• The reductionist process typical of ecological risk assessments, which extrapolates from organisms to ecosystems and from small-scale to large-scale systems, has not been tested adequately.
• The current process lacks transparency and relevant information is difficult to communicate to stakeholders.
• Little is known about how the composition of a community affects the response of organisms or ecosystems to stressors.
• Without effective diagnostic tools to link effects observed in the environment to the stressors’ mode or modes of action, it has been difficult to determine cause and effect relationships.
• Too often ecological risk assessments use metrics that result in a simplified scorecard that does not take into account the interactions of the stressors, the organisms, and the effects of the landscape; as a result, these assessments can present a misleading picture of the true impacts.

Van den Brink et al. (2016) recognized the importance of ecological risk assessment to the adaptive management process as originally proposed by Wyant et al. (1995), which explicitly incorporates social goals. Social considerations and values, as expressed by the engagement and governance process, set the management goals and limits on resources and are factored into decision making. Ideally, the science of risk assessment estimates risk, evaluates management options, lists the critical variables to be monitored, and then re-evaluates the system.

Van den Brink et al. (2016) listed 11 practical steps for improving future ecological risk assessments:

1. Build a digital map of the study site that includes land use, topography, regulatory jurisdictions, and the locations of sources, stressors, habitats, and endpoints. This map becomes the framework for the risk assessment.
2. Establish a priori the cultural and protection goals that will determine the success of the assessment and decision-making process.
3. Determine the interactions among the species and the ecological processes and functions that will be affected by the stressors. Models are recommended as the tool for codifying these interactions when building the risk assessment.
4. Map out regions in the landscape that have similar land uses, stressors, and management goals. These regions are useful in describing the distribution of risk across the study region.
5. Build a list of management activities, ranging from simple nutrient reduction to major civil engineering activities such as building cofferdams.
6. Construct a conceptual model that reflects the sources of stressors, the stressors, habitats, the expected effects, and the impacts to the system under investigation.
7. Use the conceptual model to organize all of the information that will inform the cause-effect modeling. This activity will help build the necessary model but also is a communication tool for decision makers and stakeholders.
8. Use the best tools to describe cause-effect relationships in a probabilistic manner.
9. Transform the cause-effect model into a quantitative structure using approaches that incorporate the dual deterministic and probabilistic nature of ecosystems. This recommendation explicitly recognizes that there are both deterministic and probabilistic features of ecological interactions.
10. Use large datasets and modern statistical tools to improve the accuracy of the predictions and to better quantify or reduce the uncertainty of the risk assessment process.

11. These tools are innately probabilistic and also are robust in providing evidence for cause-effect interactions.

12. Employ ecological risk assessment as part of an adaptive management framework as suggested by Wyant et al. (1995).

APPLYING ECOLOGICAL RISK ASSESSMENT TO GENETICALLY MODIFIED ORGANISMS

The development and release of genetically modified organisms brings many of the same ecological considerations as the development and potential release of gene-drive modified organisms. As such, a review of frameworks and examples of assessments that have been applied to genetically modified organisms provides useful context.

Tiedge et al. (1989) published an early summary of the potential hazards and effects of genetically modified organisms. The authors recommended that the assessment of genetically modified organisms should be based on phenotypic traits rather than on how the organism was created. They identified several factors that could be useful in estimating the effects of genetically modified organisms on the environment, including:

• Survival and reproduction of the genetically modified organism;
• Interactions between the organism and the ecosystem in which it is released;
• Effects on the structure and function of ecosystems;
• Changes in the fitness of the modified organism;
• Genetic transfer of the introduced sequence by hybridization, conjugation, transduction, or transformation; and
• Potential displacement of native species.

Recognizing the potential for genetically modified organisms to cross national boundaries, the authors suggested the need to establish a means for international coordination regarding the regulation of biotechnology.

Many of the points made by Tiedge et al. were reiterated by Snow et al. (2005) in a position paper from the Ecological Society of America. The paper, which uses an alternative term for genetically modified organisms, genetically engineered organisms (GEO), included the following conclusions and recommendations:

• GEOs should be designed to reduce environmental risks.
• More extensive studies of the environmental benefits and risks associated with GEOs are needed; effects should be evaluated relative to appropriate baseline scenarios.
• Environmental release of GEOs should be prevented if scientific knowledge about possible risks is clearly inadequate.
• In some cases, post-release monitoring will be needed to identify, manage, and mitigate environmental risks.
• Science-based regulation should subject all transgenic organisms to a similar risk assessment framework and should incorporate a cautious approach, recognizing that many environmental effects are GEO- and site-specific.
• Ecologists, agricultural scientists, molecular biologists, and others need broader training and wider collaboration to address these recommendations.

The paper is an excellent compendium of the types of genetically modified organisms, their potential uses, and the possible effects. The paper also discusses ecological risk assessment and uncertainty, though not in a concrete fashion.

Another landmark paper in the discussion of ecological effects of genetic modification is Burt (2003), which describes the use of site-specific selfish genes as tools to control natural populations. The paper discusses the probability of horizontal gene transfer and describes nuances and effects of the homing endonuclease gene, including how frequently it changes over time and its relationship to the fitness of the population. The paper’s population models are idealized, and appear to assume that an equilibrium state can be reached. These models are similar to those described in the population genetics section of the current report and draw from a framework developed originally by Hartl (1970). The paper is significant in that it covers some key considerations that inform the construction of a conceptual model and notes that the estimations of frequency change as a construct moves through a population.

By the mid-2000s, it had become apparent that the traits introduced into genetically modified plants could move to wild plants of the same species or to closely-related organisms. For example, it has been documented that the CP4 EPSPS marker, which confers resistance to glyphosphate herbicide, transferred from creeping bentgrass (Agrostis stolonifera) to sentinel plants of A. stolonifera and other Agrostis plants in the landscape; that transgenic herbicide-resistant Agrostis stolonifera had become established in areas downwind of cultivated areas, suggesting a pollen-mediated dispersal; and that Agrostis hybrids were fertile and stable (Watrud et al., 2004; Reichman et al., 2006; Kausch et al., 2010). Such examples may be useful in understanding the potential for gene flow between gene-drive modified organisms and other organisms.

Tiered approaches to assess effects have long been part of environmental toxicology and other fields. Raybould and Cooper (2005) used a series of tiered tests to evaluate the risk of changes in hybrids between virus-resistant transgenic Brassica napus and wild relatives. The authors proposed three tiers: Tier I tests for hybrid production using laboratory experiments and hand pollination; Tier II looks for spontaneous hybrids in a laboratory or field setting; and Tier III searches for naturally occurring hybridization. The authors presented case studies to demonstrate the prediction of risk using the tiered approach. However, the analysis is a comparison of exposure to an effect threshold to determine a risk quotient; as such, the description of the risk assessment and uncertainty is not quantitative, and the analysis lacks a clear conceptual model.

Wolt et al. (2010) proposed a problem formulation process that is reminiscent of the framework described in EPA’s 1998 guidance for ecological risk assessments, though the terminology used is somewhat confusing. For example, the authors state that identifying “risks of greatest relevance” is at the core of the problem formation process; however, it is not clear whether “risk” is intended to be synonymous with hazard (as is the common-language interpretation), as a probabilistic technical term, or in a discipline-specific way. In addition, uncertainty is defined as “a form or source of doubt,” which is different from its definition used in this report. Although specific to the risk assessment of genetically modified organisms, it appears that many of the authors’ key points had been superseded by earlier research.

Selecting appropriate test species is an important task in ecological risk assessment. In a review of the criteria for selecting arthropod species for testing to derive ecological risks from crops genetically modified for insect resistance, Romeis et al. (2013) identify test organisms that have been used for regulatory risk assessment. The authors recommend selecting species that are relevant and avoiding superfluous data that could distract the attention of risk assessors from more serious risks. However, the authors use the term “risk” as synonymous with hazard in this work. In addition, risk needs to be estimated before a comparative ranking of risk can be accomplished.

LEARNING FROM ASSESSMENTS UNDER THE NATIONAL ENVIRONMENTAL POLICY ACT

As discussed in Chapter 6 of this report, alterations to the environment are often assessed under the environmental assessment (EA) and environmental impact statement (EIS) process in

compliance with NEPA. Some of these assessments can provide insight into the types of environmental considerations to be included when the release of a genetically modified organism is planned as part of environmental management. The Animal and Plant Health Inspection Service (APHIS) (2008), for example, is an environmental impact assessment for the use of genetically engineered insects as part of a pest control program. As an environmental impact statement, the report does not fit the probabilistic cause-effect structure of a risk assessment. However, the report does contain information that would be useful in a problem formulation process. Section IC of the report describes a range of potential scenarios and maps the locations of rearing sites and program activities. Section III, Affected Environment, provides a detailed listing of the range of environments where the genetically modified organisms would be used. Section IIIC discusses the affected environment, including human health and non-target species. In a risk assessment, many of these lists would correspond to culturally important endpoints, whether they are cultural resources, listed species, visual resources, domestic animals, critical habitats, or wild plants or animals.

Another illustrative environmental impact assessment is APHIS (2014), which focuses on a field release of the genetically modified diamondback moth. Similar to APHIS (2008), this report does not have the probabilistic cause-effect structure found in an ecological risk assessment, but could serve as a useful resource for constructing a conceptual model and computational framework for a risk assessment of a gene-drive modified organism.

RISK ASSESSMENT FOR STERILE MODIFIED MOSQUITOES

The risk assessment conducted by Hayes et al. (2015) for a hypothetical release of a modified sterile male mosquito provides perhaps the clearest parallels to gene-drive modified organisms. The scenario features the escape of modified sterile male mosquitoes from a research facility in a setting where wild-type mosquitoes of the same species are present in the environment. There is no published experimental or field data available to incorporate into the assessment; rather, it uses fault tree models in an elaborate but well-organized expert solicitation. Because we do not have yet have field data on gene-drive modified organisms, ecological risk assessment for gene drives will likely follow a similar approach as Hayes at al. The assessment is probabilistic and addresses uncertainty, and the authors used a Monte Carlo approach to address combinations of exposures and effects. However, the endpoints do not incorporate explicit stakeholder values and are essentially only measures of exposure.

Kuzma and Rawls (2016) have recently conducted an analysis that sets the stage for the application of ecological risk assessment to gene drives. The authors emphasized the importance of engagement with stakeholders and presented the multigenerational aspect of the release of a gene drive and its ramifications both for estimating effects and creating long-term management agreements. However, it is clear from the article’s treatment of uncertainty that a great deal of specific information is missing that would have made this risk assessment more straightforward and useful for decision makers. The extensive information found in this document points to a variety of other information that may have proven useful to setting boundaries based on empirical data rather than expert elicitation. Although it addresses a non-driving modified organism and an accidental release scenario, Hayes et al. (2015) is the only risk assessment the committee could identify that follows the model put forth by Van den Brink et al. (2016).

REFERENCES

Andersen, M.C., H. Adams, B. Hope, and M. Powell. 2004b. Risk analysis for invasive species: General framework and research needs. Risk Anal. 24(4):893-900.

APHIS (Animal Plant Health Inspection Service). 2008. Use of Genetically Engineered Fruit Fly and Pink Bollworm in APHIS Plant Pest Control Programs [online]. Available at: https://www.aphis.usda.gov/plant_health/ea/downloads/eis-gen-pbw-ff.pdf [accessed May 3, 2016].

APHIS. 2014. Proposal to Permit the Field Release of Genetically Engineered Diamondback Moth in New York: Environmental Assessment [online]. Available at: https://www.aphis.usda.gov/brs/aphisdocs/13_297102r_dea.pdf [accessed May 3, 2016].

Ayre, K.K., and W.G. Landis. 2012. A Bayesian approach to landscape ecological risk assessment applied to the Upper Grande Ronde watershed, Oregon. Hum. Ecol. Risk Assess. 18(5):946-970.

Ayre, K.K., C.A. Caldwell, J. Stinson, and W.G. Landis. 2014. Analysis of regional scale risk to whirling disease in populations of Colorado and Rio Grande cutthroat trout using Bayesian belief network model. Risk Anal. 34(9):1589-1605.

Barnthouse, L.B. 2008. The strengths of the ecological risk assessment process: Linking science to decision-making. Integr. Environ. Assess. Manag. 4(3):299-305.

Bartolo, R.E., R.A. van Dam, and P. Bayliss. 2012. Regional ecological risk assessment for Australia's tropical rivers: Application of the relative risk model. Hum. Ecol. Risk Assess. 18(1):16-46.

Burt, A. 2003. Site-specific selfish genes as tools for the control and genetic engineering of natural populations. Proc. Biol. Soc. 270(1518):921-928.

Colnar, A.M., and W.G. Landis. 2007. Conceptual model development for invasive species and a regional risk assessment case study: The European green crab, Carcinus maenas, at Cherry Point, Washington. Hum. Ecol. Risk Assess. 13(1):120-155.

Dale, V.H., G.R. Biddenger, M.C. Newman, J.T. Oris, G.W. Suter, T. Thompson, T.M. Armitage, J.L. Meyer, R.M. Allen-King, G.A. Burton, P.M. Chapman, L.L. Conquest, I.J. Fernandez, W.G. Landis, L.L. Master, W.J. Mitsch, T.C. Mueller, C.F. Rabeni, A.D. Rodewald, J.G. Sanders, and I.L. van Heerden. 2008. Enhancing the ecological risk assessment process. Integr. Environ. Assess. Manag. 4(3):306-313.

EPA (US Environmental Protection Agency). 1992. Framework for Ecological Risk Assessment. EPA/630/R-92/001. Washington, DC: Risk Assessment Forum, US Environmental Protection Agency [online]. Available at: https://www.epa.gov/sites/production/files/2014-11/documents/framework_eco_assessment.pdf [accessed May 3, 2016].

EPA. 1998. Guidelines for Ecological Risk Assessment. EPA/630/R-95/002F. Washington, DC: Risk Assessment Forum, US Environmental Protection Agency [online]. Available at: https://www.epa.gov/sites/production/files/2014-11/documents/eco_risk_assessment1998.pdf [accessed May 3, 2016].

Greenberg, M., B.D. Goldstein, E. Anderson, M., Dourson, W. Landis, and D.W. North. 2015. Whither risk assessment: New challenges and opportunities a third of a century after the Red Book. Risk Anal. 35(11):1959-1968.

Hartl, D. 1970. Analysis of a general population genetic model of meiotic drive. Evolution 24(3):538-545.

Hayes, E.H., and W.G. Landis. 2004. Regional ecological risk assessment of a near shore marine environment: Cherry Point, WA. Hum. Ecol. Risk Assess. 10(2):299-325.

Hayes, K.R., S. Barry, N. Beebe, J.M. Dambacher, P. De Barro, S. Ferson, J. Ford, S. Foster, A. da Silva Goncalves, G.R. Hosack, D. Peel, and R. Thresher. 2015. Risk Assessment for Controlling Mosquito Vectors with Engineered Nucleases, Part I: Sterile Male Construct Final Report. Technical report, CSIRO Biosecurity Flagship, Hobart, Australia, 148pp.

Heenkenda, M.K., and R. Bartolo. 2015. Regional ecological risk assessment using a relative risk model: A case study of the Darwin Harbour, Darwin, Australia. Hum. Ecol. Risk Assess. 22(2):1-23.

Herring, C.E., J. Stinson, and W.G. Landis. 2015. Evaluating non-indigenous species management in a Bayesian networks derived relative risk framework for Padilla Bay, Washington. Integr. Environ. Assess. Manag. 11(4):640-652.

Hines, E.E., and W.G. Landis. 2014. Regional risk assessment of the Puyallup River Watershed and the evaluation of low impact development in meeting management goals. Integr. Environ. Assess. Manag. 10(2):269-278.

Kanwar, P., W.B. Bowden, and S. Greenhalgh. 2015. A regional ecological risk assessment of the Kaipara Harbour, New Zealand, using a relative risk model. Hum. Ecol. Risk Assess. 21(4):1123-1146.

Kapustka, L. 2008. Limitations of the current practices used to perform ecological risk assessment. Integr. Environ. Assess. Manag. 4(3):290-298.

Kausch, A.P., J. Hague, M. Oliver, L.S. Watrud. C. Mallory-Smith, V. Meier, and C.N. Stewart. 2010. Gene flow in genetically engineered perennial grasses: Lessons for modification of dedicated bioen-

ergy crops. Pp. 285-297 in Plant Biotechnology for Sustainable Production of Energy and Co-products, Biotechnology in Agriculture and Forestry, P.N. Mascia, J. Scheffran, and J.M. Widholm, eds. Berlin: Springer.

Kuzma, J., and L. Rawls. 2016. Engineering the wild: Gene drives and intergenerational equity. Jurimetrics 56(3).

Landis, W.G. 2004. Ecological risk assessment conceptual model formulation for nonindigenous species. Risk Anal. 24(4):847-858.

Landis, W.G., and J.A. Wiegers. 1997. Design considerations and a suggested approach for regional and comparative ecological risk assessment. Hum. Ecol. Risk Assess. 3(3):287-297.

Landis, W.G., and J.A. Wiegers. 2005. Introduction to the regional risk assessment using the relative risk model. Pp. 11-36 in Regional Scale Ecological Risk Assessment Using the Relative Risk Model, W.G. Landis, ed. Boca Raton, FL: CRC Press.

Landis, W.G., J.L. Durda, M.L. Brooks, P.M. Chapman, C.A. Menzie, R.G. Stahl, and J.L. Stauber. 2013. Ecological risk assessment in the context of global climate change. Environ. Toxicol. Chem. 31(1): 79-92.

Menzie, C.A., M.M. MacDonnell, and M. Mumtaz. 2007. A phased approach for assessing combined effects from multiple stressors. Environ. Health Perspect. 115(5):807-816.

Moraes, R., W.G. Landis, and S. Molander. 2002. Regional risk assessment of a Brazilian rain forest reserve. Hum. Ecol. Risk. Assess. 8(7):1779-1803.

NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the Process. Washington, DC: National Academy Press.

NRC. 2009. Science and Decisions: Advancing Risk Assessment. Washington, DC: The National Academies Press.

Raybould, A., and I. Cooper. 2005. Tiered tests to assess the environmental risk of fitness changes in hybrids between transgenic crops and wild relatives: The example of virus resistant Brassica napus. Environ. Biosafety Res. 4(3):127-140.

Reichman, J.R., L.S. Watrud, E.H. Lee, C.A. Burdick, M.A. Bollman, M.J. Storm, G.A. King, and C. Mallory-Smith. 2006. Establishment of transgenic herbicide-resistant creeping bentgrass (Agrostis stolonifera L.) in nonagronomic habitats. Mol. Ecol. 15(13):4243-4255.

Romeis, J.R., A. Raybould, F. Bigler, M.P. Candolfi, R.L. Hellmich, J.E. Huesing, and A.M. Shelton. 2013. Deriving criteria to select arthropod species for laboratory tests to assess the ecological risks from cultivating arthropod-resistant genetically engineered crops. Chemosphere 90(3):901-909.

Snow, A.A.., D.A. Andow, P. Gepts, E.M. Hallerman, A. Power, J.M. Tiedje, and L.L. Wolfenbarger. 2005. Genetically engineered organisms and the environment: Current status and recommendations. Ecol. Appl. 15(2):377-404.

Suter, G.W. 2008. Ecological risk assessment in the United States Environmental Protection Agency: A historical overview. Integr. Environ. Assess. Manag. 4(3):285-289.

Tiedge, J.M., R.K. Colwell, Y.I. Grossman, R.E. Hodson, R.E. Lenski, R.N. Mack, and P.J. Regal. 1989. The planned introduction of genetically engineered organisms: ecological considerations and recommendations. Ecology 70(2):298-315.

Van den Brink, P.J., C.B. Choung, W. Landis, M. Mayer-Pinto, V. Pettigrove, S. Scanes, R. Smith, and J. Stauber. 2016. New approaches to the ecological risk assessment of multiple stressors. Mar. Fresh. Res. 64(4)429-439.

Walker, R., W.G. Landis, and P. Brown. 2001. Developing a regional ecological risk assessment: A case study of a Tasmanian agricultural catchment. Hum. Ecol. Risk Assess. 7(2):417-439.

Watrud, L.S., E.H. Lee, A. Fairbrother, C. Burdick, R.R. Reichman, M. Bollman, M. Storm, G. King, and P.K. Van de Water. 2004. Evidence for landscape-level, pollen-mediated gene flow from genetically modified creeping bentgrass with CP4 EPSPS as a marker. Proc. Natl. Acad. Sci. 101(40):14533-14538.

Wiegers, J.K., H.M. Feder, L.S. Mortensen, D.G. Shaw, V.J. Wilson, and W.G. Landis. 1998. A regional multiple stressor rank-based ecological risk assessment for the fjord of Port Valdez, AK. Hum. Ecol. Risk Assess. 4(5):1125-1173.

Wolt, J.D., P. Keese, A. Raybould, J.W. Fitzpatrick, M. Burachik, A. Gray, S.S. Olin, J. Schiemann, M. Sears, and F. Wu. 2010. Problem formulation in the environmental risk assessment for genetically modified plants. Transgenic Res. 19(3):425-436.

Wyant, J.G., R.A. Meganck, and S.M. Ham. 1995. A planning and decision-making framework for ecological restoration. Environ. Manage. 19(6):789-796.