Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)
Chapter: Incorporation of Sensitivity Analyses - Measurement of Total Uncertainty
Incorporation of Sensitivity Analyses—Measurement of Total Uncertainty
While the measurement of the variability of microsimulation model output resulting from the sampling variability in the primary input data base is relatively straightforward, the above discussion makes clear that assessment of the variability resulting from secondary sources, such as the use of regression equations, imputations, and statistical matches can be more problematic. This is due to both theoretical and practical considerations. An example of a theoretical issue is that the sample space of alternative model specifications for, say, a behavioral equation may be very dependent on an analyst’s opinion (see below). Practically, even if the various secondary data sets used to determine various control totals, regression coefficients, etc., are fairly small and accessible, estimation of the covariance of various estimated parameters may be difficult, causing the parametric bootstrap procedure discussed above to be difficult to apply. Furthermore, since the secondary data sets may be unavailable to the analyst running the microsimulation model, if estimates of these variances and covariances are not available in the literature, their measurement may not be possible. In addition, if the input is the result of a complicated computation, access to the data set and the computational algorithm is crucial for use of the bootstrap. However, even if the various secondary data sets and procedures are available, creating bootstrap pseudosamples for each of these data sets, some of which are extremely large and have complicated sample designs, would at times involve a massive effort.
As a currently more feasible alternative to assessment of overall uncertainty, it may be possible to use sensitivity analysis in conjunction with bootstrap resampling to develop estimates of total model uncertainty. Roughly speaking, one could use the bootstrap to measure variance and a sensitivity analysis to weakly measure bias. To do this, one would bootstrap the primary database as described above. However, instead of developing a distribution of values for other inputs (such as child care expense regression equations), one could, subjectively, select, say, two of what appear to be the most important inputs and, again subjectively, develop, say, two variants of one input and three variants of the other (such as three child care expense equations). One would then have six (2×3) different parameter sets to apply to, say, four pseudosamples of the primary database for a total of 24 model runs. In this way, uncertainty due to model mispecification could be better understood. However, the resulting range would not have an associated coverage probability.
While this failure to produce a range with known coverage is troublesome, there are instances when this is the best that one can do. For many of the modules used within microsimulation models, the specific choice of which module to use is somewhat arbitrary, either because the underlying theory is uninvestigated or unclear. There is therefore a contribution to the uncertainty of the output from microsimulation models resulting from the choice of which
