Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)
Chapter: Total Variability Due to Use of Alternative Modules
variability. Third, there is the hope that superior versions of microsimulation models will have an omnibus superiority robust to choice of loss function.
We examined a variety of outputs from TRIM2. Table 2 provides definitions for these responses. The first 10 defined variables are categorical variables that were studied in both their original form and a dichotomized (0–1) version. An advantage of the dichotomized version is that, since its mean is a percentage, analysis of projections as estimates of change is simplified, there being numerous ways to analyze changes in the distribution of a categorical variable. Two outputs of great interest are the aggregate responses of total participants and total benefits.
ANALYSIS OF RESULTS
Total Variability Due to Use of Alternative Modules
The primary question we address is: What differences in the outputs result from use of alternative components in the three modules? In other words, do the alternatives make a difference? For estimates of change in the total number of AFDC participants from 1983 to 1987, the estimates from the 16 versions of TRIM2 range (in units of 1,000) from −213 to 293, while the comparison value from the quality control files is 98 (i.e., QC87−QC83=98). For estimates of change in total benefits, the estimates from the 16 versions of TRIM2 range (in millions) from $1,578 to $3,742, while the comparison value is $2,499.
These results indicate that, in using alternatives that were thought a priori to have similar success in modeling AFDC, there are versions that disagree about whether the number of participants goes up or down and by an amount that was at least twice the size of the observed change of 100,000 people. Also, these alternatives provide estimates of change in total payouts that range from increased costs of $1.6 billion to $3.7 billion. Moreover, the estimates of change are with a forecast horizon of only 4 years.
Table 3 provides other ranges for estimates of both change and level for other variables of interest. Note that the comparison value for change was outside the range provided by the 16 versions of TRIM2 for the following variables: (1) percentage of households where youngest child is less than 5 years old, (2) percentage of households where race/ethnicity of head is black non-Hispanic, and (3) percentage of households with no adult earnings. Also, it was not uncommon for versions of TRIM2 to incorrectly identify the direction of change; this occurs for 7 of the 10 variables examined. (Note that if a difference is not significantly different from 0, which may be true for some of these estimates, it is irrelevant whether the estimated sign is in the wrong direction.) For estimates of level there are also three variables for which the comparison value (as given by the QC87 value) is not included in the range of values supplied by the 16 versions of TRIM2. These are (1) percentage of
TABLE 2 Definitions of Selected Output Variables From TRIM2
|
Response Variable |
Definition |
|
Individual |
|
|
Type of unit |
AFDC unit type. The responses are basic, unemployed parent, and incapacitated parent. The dichotomized version studied was 1 for basic, 0 otherwise. |
|
Number of adults |
Adults eligible to be aided in AFDC unit. The responses are 0, 1, and 2. The dichotomized version studied was 1 for two and 0 otherwise. |
|
Number of children |
Children eligible to be aided in AFDC unit. The responses are 0, 1, 2, 3, and more than 3. The dichotomized version studied was 0 for more than two children, 1 otherwise. |
|
Number aided |
Total persons eligible to be aided in AFDC unit. The responses are 0, 1, 2, 3, 4, and more than 4. The dichotomized version studied was 0 for more than three, 1 otherwise. |
|
Age of youngest child |
Age of youngest child in AFDC unit. The responses are 0–1, 2–4, 5–9, 10–14, and 15–18. The dichotomized version studied was 1 for less than 5, 0 otherwise. |
|
Race of head |
Race/ethnicity of AFDC unit head. The responses are white non-Hispanic, black non-Hispanic, Hispanic, and other non-white. The dichotomized version studied was 1 for black non-Hispanic, 0 otherwise. |
|
Adult earnings |
Monthly earned income of adults in AFDC unit in 1987 dollars. The responses are none, $1–$199, $200–$499, and more than $499. The dichotomized version studied was 0 for 0 earnings, 1 for positive earnings. |
|
Marital status |
Marital status of AFDC unit head. The responses are never married, married spouse present, married spouse absent or separated, married spouse in the armed forces, married spouse absent (other), widowed, and divorced. The dichotomized version studied was 1 for no spouse, 0 otherwise. |
|
Sex of head |
Sex of AFDC unit head. Female was coded as 1, male as 0. |
|
Age of head |
Age of AFDC unit head. The responses are 14–16, 17–19, 20–24, 25–29, 30–39, 40–49, 50–59, and 60 and over. The dichotomized version was 1 for less than 20, 0 otherwise. |
|
Gross income |
Gross monthly income of AFDC unit members in 1987 dollars. The responses are none, $1–$199, $200–$499, and more than $499. |
|
Relationship |
Relationship of AFDC unit head to household head. The responses are head/spouse and other. |
|
Size of benefit |
Size of benefit in 1987 dollars. |
|
Aggregate |
|
|
Total participants |
Average monthly number of AFDC participants (in thousands). |
|
Total participants with earnings |
Average monthly number of AFDC participants with earnings (in thousands) |
|
Benefits |
Annual benefits (in $ million) to AFDC participants. |
TABLE 3 Observed Differences for Various Estimates from TRIM2 Validation Experiment
households where there are more than two children, (2) percentage of households where the number receiving assistance is greater than three, and (3) percentage of housholds where the head of household is less than 20 years old. In general, Table 3 demonstrates that potentially important differences were obtained for many of the responses studied when alternate modules were used, although the actual policy implications of many of the differences depend on the intended application of the estimates.
These observations are not meant to imply any weakness of TRIM2 relative to other models, microsimulation or not, whose goal is to provide estimates of the above quantities. No such conclusions can be drawn, given that this analysis does not include other modeling approaches. What this experiment does demonstrate is the large amount of variability that results from reasonable changes to the basic model.4 In other words, the alternatives that were investigated made a difference.
