9
Evaluating the Accuracy of U.S. Population Projection Models
Laurence Grummer-Strawn and Thomas J.Espenshade
INTRODUCTION
Validation experiments of projections from microsimulation models have been relatively rare. Many reasons have been offered for this situation, including the complex nature and multiplicity of microsimulation calculations, the generally low priority given to assessments of error and sources of uncertainty, and the time pressures that usually surround policy debates and analyses. Perhaps more important than any of these, though, is the fact that microsimulation projections are typically conditional forecasts; they are answers to a series of “what if” questions. If none of the policy options being evaluated is ever adopted, it is difficult to validate and analyze the projected associated policy consequences. Moreover, as noted in Part I of Volume I, even if an analyzed policy is enacted in “pure” form, disentangling the multiple specific sources of potential error is often problematic. Finally, the policy process is inevitably handicapped in its ability to generate the kinds of information that would make validation possible on a regular basis. Once a particular program is adopted, for example, the program’s features may be altered only occasionally, which makes it difficult to trace observed policy outcomes back to specific program elements.
Laurence Grummer-Strawn and Thomas J.Espenshade are at the Office of Population Research, Princeton University; the latter served as a member of the Panel to Evaluate Microsimulation Models for Social Welfare Programs. The helpful comments of Eugene Ericksen, the generous cooperation of Alice Wade, and the technical assistance of Diane Van Houten are gratefully acknowledged.
Population projections also are conditional forecasts; they, too, respond to a series of “what if” questions. Their products are projections of future total population size, rates of growth, and population composition by age, sex, race, and (sometimes) marital status, conditional on a set of specific assumptions about future patterns of fertility, mortality, and migration. Our review of the literature suggests, however, that, while not cornucopian, attempts to validate population projections are neither as rare nor as difficult as those stemming from microsimulation models.
Population projections are far simpler than microsimulation models. First, cohort component models that are typically used to perform population projections are not behavioral models, thereby greatly simplifying the task. Instead they are mechanical deterministic models that capture the dynamic processes by which populations grow, age, and alter other features. Second, population projection models have only four ingredients. In addition to a baseline population that must be specified, assumptions about future trends in fertility, mortality, and migration are the only other required inputs. Third, in contrast to microsimulation models, cohort component population projection models are macrosimulation models (more precisely, cell-based models), implying drastic simplifications in both the number and the transparency of calculations performed.
This chapter is concerned with two broad aspects of evaluating the accuracy of population projection models. First, we are interested in what can be learned from efforts to validate population projections in general, especially those performed with the aid of cohort component methods, regardless of whether the projections pertain specifically to the United States. Our aim is to illustrate the variety of methodologies that have been used to evaluate population projections and to highlight some of the relevant findings. Second, particular attention is focused on projections of the U.S. population prepared by the Bureau of the Census and the Social Security Administration’s (SSA) Office of the Actuary. These projections are especially relevant because they are frequently used to calibrate microsimulation models.
The nature of cohort component population projections is described in the next section. This is followed by a review of alternative validation strategies and outcomes for population projection models, including sensitivity analysis, external validation, and use of confidence intervals to assess variability in population forecasts. Our discussion draws on existing evaluations of population projection models as much as possible, but these are supplemented by our own analyses when feasible.
