CRAIG TAYLOR, ERIK VANMARCKE AND JIM DAVIS

WHAT ARE MODELS OF RISKS from natural hazards? What functions do they serve? How can they be evaluated and tested? What is the current status of these models? Do they serve the purposes intended? Can we trust them?

These questions arise in a wide variety of contexts, but of special interest are those contexts in which public concern is expressed over the degree of confidence that should be placed in various models of risks from natural hazards and in the results that these models produce. The use of risk models—often with proprietary components—for hurricane and earthquake rate-making often produces heated controversies among scientists, engineers, and the public at large (Dennis Kuzak, EQECAT, personal communication, 1996; Jack E. Nicholson, Chief Operating Officer, Florida Hurricane Catastrophe Fund, personal communication, 1996). For instance, many questions have arisen about the contrast between two rates based on proprietary models: the very low rates established in the original California earthquake insurance program (under the FAIR Plan) and the rates that now appear very high to the public in the new California

Earthquake Authority (CEA) insurance program. (For more details on the CEA, see Chapter 4.) Is there an acceptable level of consistency and validity in how the rates are developed? Is there a sound statistical basis for the development of the rates? Do alternative assumptions used in risk analysis techniques lead to either lower or higher rates? From an insurance perspective, do the risk analysis techniques used integrate with reinsurance pricing and costs? How can the public have confidence in rate-making procedures if these are proprietary? While applying modeling to catastrophe insurance issues has received much publicity, licensing considerations, standardization requirements, cost-benefit studies, risk reduction prioritization procedures, and tools for emergency response planning have also raised significant issues about the application of new risk technologies to social problems.

This appendix first outlines what we mean by models of risks from natural hazards. Then, we identify some of the functions of these models and address issues involved in evaluating and testing these models, such as their trustworthiness considering their proprietary nature when used for public purposes. We do not address in detail how these risk models are constructed, although evaluation and testing of models obviously requires intimate knowledge of this process.

From a public standpoint, evaluation testing and standardization procedures are necessary when the models are employed to set rates or regulations in the public interest. At present no consensus exists on models produced in the private sector. In fact, rigorous evaluations of models have to date produced varying results. CEA hearings in California and activities of the Florida Commission on Hurricane Loss Project Methodology have illustrated the varying results of disciplined queries into current models. (See State Board of Administration of Florida, 1996, for examples.) There are questions at present as to how these evaluation and testing procedures—as well as the development of guidelines, criteria, and standards—can be most effective and equitable in serving public and private needs. Attempts currently under way to evaluate and test proposed risk models could better define their appropriate uses and limitations. To the extent possible, given that proprietary elements in modeling may be legitimate in some cases, these efforts can build successfully on past endeavors to evaluate theories, data, and projections.

To respond to this question, we propose some formal characterizations of models of risks from natural hazards, generally consistent with discussions in EERI (1984) and Sauter (1996). These formal characterizations can be supplemented with a historical approach to natural hazards model developments. Models should be understood in relationship to the issues that they are designed to address. Sometimes models focus on very specific practical or theoretical issues, but are then used to address other issues for which they are ill suited.

A model is a general representation of reality. The reality represented may be physical, as in the case of a natural hazard, or it may include significant human elements, as in a model of damage to a building or of a financial risk. A model may be used to guide our understanding and action with respect to the phenomena that it represents. Modeling is an activity associated with scientific, engineering, economic, financial, or more generally technical activities. Models can be valuable as tools to place constraints on, say, the dimensions of losses owing to natural hazards. But not everything that we say or think can be expressed by a model. There may not be a model for the salvage value of real estate after a landslide because there is only anecdotal information on the impact of such events on property value. Most landslides, we expect, greatly harm the value of the real estate that they affect. Some landslides may have cleared away old structures which are typically vulnerable to earthquakes and extreme winds, and in rare cases may have stabilized the slope, enabling developers to construct new and less vulnerable buildings in prime real estate areas. This may be worth looking into. However, as long as we rely only on anecdotal and conjectural information, we do not have much of a model.

Models are often primarily qualitative. This is true, for instance, of models of political theory, law, organizational behavior, or even models in the fields of biology and geology. In spite of the primarily qualitative nature of these models, they may still be buttressed by quantitative information. Thus, case studies of how organizations function may be based on quantitative data, but the models of organizations constructed from the data may be largely qualitative. Micro-zoning natural hazards for land use planning may be based on qualitative as well as quantitative models, but even the qualitative models are likely to be supported by quantitative information.

Quantitative models vary in the degree to which they are deterministic as opposed to statistical or probabilistic. Many models in physics are primarily deterministic, in the form of a ''law" or assertion that some phenomena are universally valid. A deterministic forecast of the tides at Laguna Beach will, for example, postulate their predicted height at some point of time in the future, and will ignore variations in this estimate. In contrast, statistical or probabilistic models incorporate the notion of variations and possibly the factors that cause or influence these variations.

Models may be deterministic in most respects, but still contain random variables or functions, or various statistical or probabilistic considerations. For instance, some early models of earthquake insurance risks incorporated many statistical elements in order to arrive at a 90th percentile estimate of a loss from a maximum credible earthquake. These estimates are deterministic insofar as they are based on a single scenario, but they may utilize statistical elements when providing damage ranges. Real-time models of hurricanes, flooding, tornadoes, or earthquakes may likewise be based on a single scenario, but have probabilistic features when quantifying the variability in forecasting the impacts of these deterministic events.

Current models of earthquake insurance risks are primarily statistical, yielding probability distributions of losses obtained from a representative ensemble of scenarios. Some early models of hurricane risk began in a similar fashion by modeling loss distributions (Friedman, 1974). Current hurricane models have built on this earlier tradition.

A model of risks from natural hazards is a general representation of the chances of damage, disruption, loss, injury, death, or contamination as a result of one or more natural hazards potentially acting on people, structures and other facilities, and/or the environment. A comprehensive model of a risk from a natural hazard generally requires many models and submodels such as models of exposures (of persons, human works, the environment), natural hazards (their character, distribution, frequency, and their local physical effects), vulnerabilities (the proneness to varying degrees of loss or disfunctionality relative to local physical effects or severities), and risk calculations (the systematic use of the other component models in order to provide quantitative information on risk).

Modeling risks from natural hazards is actually a specialized field within science and engineering. Most scientists and engineers working in this area focus on some portion of the overall risk problem, such as how to design steel buildings to withstand seismic and wind loads or how to

identify and evaluate landslide potential. Even though the history of efforts to model risks from natural hazards is unfortunately sparse, insights into various challenges and developments in risk assessment for natural hazards are provided by Freeman (1932), Friedman (1974), Petak and Atkisson (1982), Steinbrugge (1982), and Eguchi and others (1989).

It is desirable to supplement formal characterizations of models of risks from natural hazards not only with an account of the history of modeling efforts, but also with an understanding of the purposes of the models. Typically, models involving natural hazards are designed to solve one specific issue or set of issues. As a consequence, the models involving natural hazards designed for some limited purpose cannot be readily adapted to other purposes. For instance, models used in the design of resistant buildings, the qualification of nuclear power equipment for licensing, the remediation of soils, or the designation of landslide zones for initial screening purposes, all typically seek to quantify or justify various "safety factors." When such built-in factors of safety or conservatism are part of models used in insurance and reinsurance, practical results may be biased and impact the pricing, availability, and afford-ability of coverage. Similar remarks apply to the use of design models for regulatory efforts, such as the development of cost-benefit ratios. To understand models for natural hazards, it is thus desirable to understand the specific and limited purposes for which they are used.

Natural hazard risk models have a large number and variety of uses in insurance and government. They can serve as decision support systems for:

• emergency response planning;

• real-time emergency actions such as evacuation;

• real-time response and recovery activities such as determining federal and state disaster assistance, and relocating displaced households or managing claims adjustment activities;

• pre-disaster land use planning such as avoidance of various hazardous sites or requiring further evaluation and possibly geotechnical remediation for these sites;

• pre-disaster preventive actions, such as the choice of building codes and building code design levels, and their enforcement;

• developing regulatory criteria for assessing whether insurers are

over-committed with respect to their coverages of risks from natural hazards;

• developing regulatory criteria for assessing the suitability of private insurer rates for natural hazards risk coverages;

• evaluating the effectiveness of public insurance programs to cover risks from natural hazards;

• developing estimates of pure premiums and loss exceedance probabilities for use in actuarially based premiums and in financial planning;

• developing underwriting guidelines or other strategic underwriting measures;

• selecting how much reinsurance to purchase.

These models should be evaluated in relation to the uses for which they are designed. Those that are adapted for one use may be poorly adapted for others. For instance, models for evaluating and updating real-time losses are probably ill-adapted to the evaluation of building codes and building design levels. Greater attention needs to be paid as to whether or not the quantitative results of models, however accurate, serve the needs of the decision makers. The principal function of models of risks from natural hazards is to systematize and integrate considerable diverse information into a format that is useful in decision making. The issue of whether models serve this purpose, and whether the criteria used are appropriate, remains to be evaluated.

The nature of the uncertainties in analyses of risks from natural hazards has long been considered important and now is receiving a great deal of attention. (See, for example, Abrahamson et al., 1990, for a provocative discussion of the distinction between random and epistemic uncertainties; and the National Research Council, 1997, for a critical review of this distinction. For in-depth discussions of the development of probability theories, and the philosophical issues to which they give rise, see Will, 1974.) Currently, concern over the new application of risk-based models for earthquakes and hurricane insurance rate-making has led to an increased focus on the question of how accurate they are. Can these models be used to develop very precise rates? These models probably cannot "incorporate" all uncertainties in some quantitative

fashion because of their complexity and the uneven development of the technologies that underlie them. The new application of risk tools in rate-making has focused attention on how public officials should respond to the uncertainties inherent in these models.

One characteristic of very complex models, such as those developed for hurricanes and earthquakes, is their uneven development. For some of the submodels, there may be very substantial databases that give them a high degree of reliability. For instance, some underwriters maintain a meticulous record of their exposures based on standardized inspections and the use of standardized policy forms. For some submodels, such as how seismic wave amplitudes decrease as seismic waves move through the earth's crust, substantial databases will reduce but not eliminate the variability in estimates. In other instances, submodels are highly theoretical or subjective, based on almost no data such as the modeling of business interruption or the effects of retroactive building codes on post-disaster repair costs. (See Eguchi et al., 1989; Morrison, 1985; Taylor et al., 1998, for descriptions of the problems of applying linear models to capture systemic impacts, such as business interruption, and the problems in applying oversimplified subjective models of direct damage-ability.) For some submodels, any number of competing approaches may be equally valid, considering the degree of uncertainty surrounding the estimates. To the extent that this is true, models of risks from natural hazards may be more suitable for indicating trends than for providing very precise results.

How then does one evaluate and test natural hazards models when they are in a state of flux and uneven development? For exposition purposes, we first offer some comments on various forms of testing and validation . We then consider briefly the common method of corroboration. Finally, we consider the development of guidelines, criteria, and standards , which some government officials have in the past resorted to in order to assure that there is a fair playing field and that the performance of facilities subjected to natural hazards can be compared against certain nominal or minimally acceptable standards.

Given the unevenness of information available for developing accurate component models or submodels, one means to guide testing and validation is to perform sensitivity analyses. These analyses indicate whether the uncertainties in submodels make much of a difference to the overall risk estimates produced. These analyses can yield surprising results. Sometimes improving a submodel will not matter much in spite of the limitations of the information on which it is based. For instance, a

precise estimate of the salvage value of real estate in a landslide region may not be needed in order to justify in cost-benefit terms a community program to avoid development in landslide-prone regions. Sensitivity analyses can thus serve to focus on those uncertainties that make greater differences to overall model results.

In further preparation for testing and validation, public agencies can themselves gather pertinent data that can be used in models. The United States Geological Survey, state geological agencies, the National Oceanic and Atmospheric Administration, and the National Flood Insurance Program are among the agencies that provide data that can be used in risk assessment for natural hazards. The California Office of Emergency Services, supported indirectly through the Federal Emergency Management Agency, has spearheaded efforts to accumulate loss and damage data from the 1994 Northridge earthquake. The California Department of Insurance has taken the lead in compiling earthquake insurance loss data so necessary to the development of earthquake loss models.

Not only do these institutionalized data developments assist in improving existing models, but public data sources can also serve as benchmarks in assessing the validity of models. For example, public data on wind velocities can assist regulators in assessing whether existing hurricane models adequately quantify wind velocities, an important element in the modeling process. By using these data as benchmarks, regulators can thus provide greater assurance to the public that modeling efforts have been critically scrutinized (Jack E. Nicholson, FHCF, personal communication, 1996).

Actual testing and validation procedures are well understood in principle although sometimes complex and difficult to implement. They include quality assurance on all submodels, as well as evaluation of the relevance of the models for the problems at hand. Such testing and validation can be very complicated and require considerable data and expertise. How, for instance, can one distinguish between a forecast that is lucky (e.g., comes out about right in one event, as a result of offsetting errors) and one that is off by two standard errors as would be expected in some percentage of cases? How can one determine that the statistical and numerical procedures used are sound? How can one show that the submodels are based on the state of the art?

Since testing and validation require considerable expertise, one may be inclined to use corroboration as a means of evaluating natural hazards models. In professional circles, the use of peer review is common, and can signify to the general public that at least a second opinion has

been sought. Since second opinions can also be flawed or biased, another proposal is the construction of guidelines, criteria, or standards for the adequacy of specific natural hazards models (Jack E. Nicholson, FHCF, personal communication, 1996; State Board of Administration of Florida, 1996).

Guidelines are suggestions for future action but may eventually be modified into criteria. Criteria are products of scientific inquiry, not the result of a process where personal values, negotiation, and compromise guide the results. They are expressions of existing knowledge and may be only feeble representations of the truth when knowledge is incomplete. If properly prepared, criteria should be useful to legislators, policymakers, and citizens as scientifically defensible and unbiased sources of information about what is known.

Criteria represent a general agreement or consensus which should be tied to documents specifying the current state of knowledge or technology. Criteria also serve as a basis for continuing scientific inquiry and research and provide decision makers with information to weigh costs and benefits associated with alternative strategies and objectives (Atkisson and Gaines, 1970; Faith and Atkisson, 1972).

Standards, on the other hand, are the product of policy and political considerations. Whereas criteria are descriptive or predictive, standards are prescriptive in that performance (code) levels must be legally met. The adoption of standards based upon criteria established as a result of codifying the state of knowledge should govern the performance of models (Atkisson and Gaines, 1970; Faith and Atkisson, 1972).

Using guidelines, criteria, or standards overcomes the ad hoc limitations of validation and testing on a case-by-case basis. One knotty problem currently faced by regulators is how to assess proprietary hurricane and earthquake models used by insurance companies. Should these models be licensed? Should insurers be allowed to use them for setting specific rates (or even in setting very rough rating guidelines)? At what point and to what extent should regulators and the general public have access to the details of these proprietary models? The use of guidelines, criteria, or standards assists the regulator in addressing these difficult issues by their consideration of both public and private interests. There is some urgency in taking this step when risk models are used to develop homeowners' insurance rates and these homeowners lack confidence in how their rates are determined.

The effectiveness of the use of guidelines, criteria, and standards can be illustrated by the practice of licensing professionals working in the

geosciences. (Over half the states have these licensing requirements and guidelines for geologists, engineering geologists, and/or geophysicists.) In California in 1969, registration was required for minimum qualification for practicing geology and geophysics as it impacts safety. Since that time, standards have been refined, and specialties such as engineering, geology, and hydrology have been added. This licensing practice received even more attention after 1989. Some practitioners have decided to cease practice and others have been brought before the Professional Registration Boards.

Another example of the challenges of implementing guidelines or standards is Freddie Mac's recent earthquake requirements for new loans, which has led to a reaction embodied in the signing of Senate Bill No. 1327 (Section 10089.4 of the Insurance Code) in California. This bill in turn has led to questions on how to operationalize the following statutory language:

The foregoing language confirms that the public requires greater confidence than now exists in the new applications of models of risks from natural hazards to the development of detailed mortgage underwriting guidelines.

The development of criteria, guidelines, and standards thus promises to create a fair playing field in how risk models for natural hazards are used. How urgent it is to develop these criteria, guidelines, and standards depends on the degree to which decisions impacting the general public are being made based on these risk models. If the experience with regulation in the geosciences is any indication, the development of these criteria, guidelines, and standards will take considerable time, with potentially disappointing results, especially during periods when public awareness is diverted to other issues.

Models of risks from natural hazards are general representations of how disasters can cause damage, loss, injury, death, or environmental impairment. As such, these models, if tested and validated, can serve to guide a variety of public and private actions in insurance and government. Recent public concern over the uses of some of these models raises the issue of how public officials should respond. In particular, what ways will effectively assure that public interests are served without interfering with private markets?

Questions about testing, validation, guidelines, criteria, and standards for natural hazards that have clear public impacts are significant because there is in general no current consensus on natural hazards models. Various attempts to test models in terms of their prediction of actual events, through simulations and their scientific and engineering bases, have not yielded any consensus. One reason for this has been the very complexity of the models themselves, some of which could, without harming available evidence, be replaced by any number of competing quantitative models. Private vendors have also been reluctant to divulge trade secrets.

The state of knowledge, which is the basis for specifying criteria, must be open and available. The criteria-setting process, in turn, is the basis for the development of standards that in the judgment of government administrators allow for appropriate margins of safety and public welfare. Recent rate hearings have resulted in a compromise that proposed rate increases be supported by adequate evidence for the increase. This compromise has not, to date, required full disclosure of private models. Even more recently, in S.B. 1327 the California State Geologist must be conferred with on mortgage-lending requirements based on private models.

These processes are in a state of flux. As a result, we must look to the future for the development of guidelines, criteria, and standards and await further tests to determine the validity of the many models that exist to assess risks from natural hazards. Ongoing public efforts, however, could continue to illuminate the appropriate uses and limitations of these existing models.