Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)
Chapter: Need for Further Validation
effect of the use of one or the other routine on estimates of level for 1987, while the current routine performed slightly better for estimates of change in the caseload from 1983 to 1987.10 No comparisons were performed of characteristics of the component of the caseload with earnings.
Finally, Long (1990) carried out an external validation study of the ability of the current, more complex TRIM2 MONTHS routine to replicate the monthly employment and income patterns evidenced for calendar year 1985. As measures of truth, the study used data from the 1984 SIPP panel for people in the labor force at any time during 1985. Long found a number of significant differences between the monthly patterns produced by MONTHS and the patterns evident in SIPP. The MONTHS routine produced a roughly similar distribution of labor force activities by month to that of SIPP, but the differences were much larger for subgroups of the population. For example, MONTHS assigned significantly more mid-year employment to women and significantly less mid-year employment to teenagers. The MONTHS routine tended to allocate greater numbers of spells of labor force activities over the year than were reported in SIPP. In particular, a high proportion of people who reported two spells in SIPP were assigned three or more spells by MONTHS. This difference primarily resulted from the MONTHS procedure that wraps periods of labor force activities that are randomly selected to begin late in the year around to the beginning of the year. Because MONTHS treats each month as containing 4.333 weeks, whereas SIPP assumes either 4 or 5 weeks per month, MONTHS showed a more even distribution of labor force activities and earnings across the year compared with SIPP. The MONTHS routine also showed a more even distribution across the year of receipt of unemployment insurance, workers’ compensation, and all other income than was evident in SIPP. Long makes a number of recommendations for modifications to the current MONTHS routine; however, the study does not attempt to estimate what effect those changes might have on the quality of the estimates produced by the TRIM2 model.
Need for Further Validation
Clearly, there are differences in the way in which each of the three models approaches the task of assigning monthly employment status and monthly income streams based on the largely annual data available in the CPS. These differences may well have an impact on the model results. (Some important differences were found simply by comparing old and current months routines
|
|
for a single model.) A useful exercise would be to conduct the same set is not clear whether the problem lies solely with errors in the CPS or also involves errors in the case records. |
