Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)
Chapter: DESIGN OF EXPERIMENT ISSUES
owing to the likely correlation between states for some response variables. The limitations that result from examining TRIM2 at one point in time help demonstrate that model validation should not be an occasional examination of a model. Rather, it should be a continuous process that accumulates knowledge about potential weaknesses, the size of errors, and situations in which the model is fallible.
Another limitation of the present experiment was that, in addition to simulating just one time period, just one change in the law was simulated. Had time and resources permitted, we would like to have simulated alternative policy scenarios—for example, a mandated AFDC minimum benefit nationwide, which would have represented a major policy change. Of course, we could not have conducted an external validation of alternative policy scenarios because they were never enacted. We could have conducted sensitivity analyses of alternative policies and obtained information relevant to the question of whether, in fact, microsimulation model estimates of differences between two policies are less variable than estimates for a particular policy because of sources of error similarly affecting both policies.
A final limitation of the experiment was that we examined only one model, owing to the time and resource constraints that we, the sponsoring agencies, and the agencies’ contractors operated under. It would have been desirable to expand the experiment to include other major models in use today, such as MATH, HITSM, DYNASIM2 (Dynamic Simulation of Income Model 2), PRISM (Pension and Retirement Income Simulation Model), and MRPIS (Multi-Regional Policy Impact Simulation) model. We should note, however, that all of these models are not directly comparable, because they are often used for different purposes and, by design, cannot necessarily produce estimates of the same quantities. Therefore, adding the model class as another factor in, say, an analysis of variance, may not generally be feasible. However, in cases in which models are directly comparable, at least for some estimates, one could certainly expand our basic experiment in this direction.
If the problems of comparability could be overcome, there are major advantages to be gained through comparing the effectiveness of different global modeling strategies. For example, the question of the relative advantages of static versus dynamic aging could be addressed in this way. Comparing several models with the truth (or a reasonable surrogate), along with a comprehensive analysis of model differences as exemplified by Haveman and Lacker (1984), will often yield great insight into the strengths and weaknesses of the various models.
DESIGN OF EXPERIMENT ISSUES
As model validation of microsimulation models is pursued further, some issues of experimental design that were ignored for the most part here will have to be
