ignores the structure implicit in the 16 model versions. One way to avoid this is to use robust analysis of variance on the discrepancies, standardized to have roughly the same units, where the analysis of variance would be a 2×2×4 design with 13 replications per cell for the 13 different response variables.

Another problem with the above analysis is that large discrepancies cannot be distinguished from moderately large ones. This raises the general issue of the modeling goal and use of omnibus loss functions. A question that must be addressed (possibly repeatedly every few years) is whether it is desirable to have a model that predicts everything equally well or whether there are some responses that are more crucial than others. The answer, of course, determines which responses play a role in the analysis and the degree to which they are weighted. In addition, a metric on the errors must be chosen that compares errors of various magnitudes, so that it can be declared how much more disturbing 10 percent errors are, say, than 5 percent errors for each response. If a useful loss function can be identified, the implied weights could be used in a weighted analysis for most of the methods discussed here.

One of the major advantages of microsimulation models is that they provide information on the distributional impacts of changes in social welfare programs, generally unavailable from other forms of modeling. Up to now, we have analyzed only the categorical data in their dichotomized version, which was used to facilitate analysis of change. There are certainly situations where a single category, or a collection of related categories, is of primary interest, and in those cases dichotomizing so that the percentage of cases in that category (or categories) is analyzed is appropriate. However, at other times the full distribution is of interest. We therefore examined as a continuation of the external validation of TRIM2, for the undichotomized frequency table outputs and for estimates of level, how close the output frequencies from the various versions of TRIM2 corresponded to those from the 1987 quality control data. Table 7 presents the χ² test of independence for a 2×r contingency table, where r is the number of categories in the response, in which one row contains the frequencies from one model version and the second row contains those from the quality control data for 1987. Notationally, the test is as follows:

where n_ij is the number of persons in category j estimated by run i; n_i. is the sum over categories—the total number of people “produced” by run i; n._j is the sum over runs—the total number of people in a certain category; and N is the total number of people “produced” by the two runs (in which one run, in this case, involves the quality control comparison values). The quantity Q

has a chi-square distribution with r−1 degrees of freedom. Values of Q are provided in Table 7 for all 16 model versions and for the 1983 quality control data for a variety of model outputs.

In examining Table 7, as in the analysis of change, it is seen that no particular model version has any noticeable advantage over the other versions. Clearly, all model versions perform well in projecting the distribution of the relationship of unit head to household head, number of children, and race of head of unit. On the other hand, all model versions perform poorly for gross income of unit. There are some specific findings that remain to be confirmed. One is that adjustment appears to be useful to match the distribution of the comparison values for age of the youngest child. Full aging appears to be useful for sex of head, but not useful for marital status. This type of detailed analysis is clearly only suggestive and can be overdone since the study is

limited. However, it is clear that the variability attributed to the choice of response variable dominates the variability due to model versions.

The last column of Table 7 displays the results from using the 1983 quality control data, which as mentioned above is not a completely fair comparison given the size of the policy change examined. Nevertheless, the 1983 quality control data do not compete well with the 16 versions of TRIM2 in the analysis shown in Table 7. However, they do outperform many TRIM2 versions in estimating gross income of unit and relationship of unit head to household head.

This test can also be used for more than two rows to investigate the similarity of several model versions as part of an analysis of the sensitivity of the distributions to model structure. For a categorical response with four categories, one could form the 16×4 contingency table and evaluate Q. However, this analysis would ignore the special structure that the 16 models have. There is no entirely satisfactory way, currently, to handle what is essentially an analysis of variance of frequency distributions. One can use dichotomization, as done above. Another way of partially circumventing the problem is to separately analyze particular subsets of the 16 model versions by using the test for independence and to look for similarities and differences in the separate analyses.