virtually no studies that attempt to validate participation functions externally through comparing simulations of program changes with actual outcomes¹⁴ or through ex post forecasting of current law with an earlier database, as was done in the panel’s validation experiment. The panel considered including alternative participation functions in its experiment, but for several reasons did not pursue this idea.

In the course of conducting its experiment, the panel did, however, learn a good deal about the current simulation and calibration of AFDC participation in TRIM2 (see the appendix to this chapter). The problems that came to light certainly support the need to perform validation studies of the participation function. They also support the need to conduct in-depth investigations of the quality of the March CPS and the administrative program data.

Two baseline files examined in the course of the validation experiment conducted by the Panel to Evaluate Microsimulation Models for Social Welfare Programs (see Cohen et al., Chapter 8, in this volume) provide information about the performance of the TRIM2 simulation of participation in the AFDC program and calibration to administrative targets. These files were a 1983 baseline produced from the March 1984 CPS and a 1987 baseline produced from the March 1988 CPS. With regard to the variables used as controls, application of the TRIM2 AFDC participation and calibration functions produced good results for national and state estimates of participating units, particularly in 1983. However, the results were less satisfactory for estimates of the percentage of the national caseload with earnings. Specifically,

• Total simulated participating units were 99.4 and 99.9 percent, respectively, of the administrative control numbers for 1987 and 1983.

• In 1987 simulated participants were within plus or minus 1.9 percent of the administrative number in 24 states, including 6 of the 10 largest states, and within plus or minus 3.9 percent in 41 states, including 8 of the 10 largest; in only 2 states did the simulated number differ from the target by 10 percent or more.

• In 1983 simulated participants were within plus or minus 1.9 percent of the administrative number in 34 states, including all of the 10 largest, and within plus or minus 3.9 percent in 45 states; in no state did the simulated number differ from the target by 8 percent or more.

• However, simulated participants included 12.7 and 7.8 percent, respectively, with earnings in 1987 and 1983 compared with 7.7 and 5.4 percent in the administrative data for those 2 years (expressed another way, simulated participants with earnings were 164 and 145 percent, respectively, of actual participants with earnings in 1987 and 1983).

The explanation for these results has to do with discrepancies between the simulated data on eligible units and the administrative data on recipients. First, the simulated eligible population from the March CPS invariably contains a higher proportion with earnings than are shown in the administrative data—15 percent versus 8 percent in 1987 and 13 percent versus 5 percent in 1983. These differences may stem from errors in the March CPS (see Citro in this volume), such as greater undercoverage of welfare-eligible groups, which are less likely to have earnings, and also from errors in the administrative data, such as underreporting of earnings. Some part of the difference may be accurate in that eligible units with earnings may be less likely to apply for AFDC benefits than other units.

Second, simulation of eligibility by state invariably results in several states with fewer simulated eligible units than reported participants. Again, population undercoverage may in part account for this finding. Moreover, apparently due in large part to sampling variability, the number of such states varies across years. In 1987 there were eight states with fewer simulated eligible units than participants; in 1983 there were only two such states.¹⁵ (Examination of other baseline files showed three such states in 1982, eight in 1985, and five in 1986; see Giannarelli [1990].)

The calibration process endeavored to satisfy the constraints both for participants in each state and for the proportion with earnings. To match state-level controls for states with high simulated participation rates, it was necessary to select all or most of the eligible units in these states. Inevitably, this meant that the simulated participating caseload in these states included a higher proportion of units with earnings than shown by the administrative data; consequently, the national controls for participants with earnings could not be matched. The problem was more severe in 1987 than 1983 because of the larger number of states with very high participation rates.

For variables not used as controls, application of the AFDC participation function and calibration process produced mixed results. Thus, total dollars of AFDC benefits are not part of the calibration, and, in fact, the 1987 and 1983 calibrated baseline simulations do not closely match the profile of benefits by state:

• The total dollar benefits simulated represented 95.1 and 97.3 percent, respectively, of actual benefits paid out in 1987 and 1983. Both percentages are lower than the corresponding percentages of simulated to actual participating units, and the difference is greater for 1987, which had a higher proportion of simulated participants with earnings whose benefits on average are lower than those of other participants.

• The state profile of the difference between actual and simulated benefits is very flat in 1987. After correcting for the national underestimate of benefits, simulated benefits were within plus or minus 1.9 percent of actual benefits in only 9 states, including 2 of the 10 largest, and within plus or minus 3.9 percent in only 15 states, including 2 of the largest. In 19 states simulated benefits differed from actual benefits by plus or minus 10 percent or more, and in 11 states simulated benefits differed from actual benefits by plus or minus 15 percent or more.

• The state profile of the difference between actual and simulated benefits is flat in 1983 as well. After correction for the national underestimate of benefits, simulated benefits were within plus or minus 1.9 percent of actual benefits in only 8 states, including 4 of the 10 largest, and within plus or minus 3.9 percent in only 18 states, including 5 of the largest. In 13 states simulated benefits differed from actual benefits by plus or minus 10 percent or more, and in 7 states simulated benefits differed from actual benefits by plus or minus 15 percent or more.

Tabulations of various characteristics at the national level showed that the calibration was reasonably successful in reproducing the profile of the caseload shown in administrative records in many, but not all, instances. The problem for the calibration is that the simulation of AFDC eligibility from the March CPS produces two distinct groups: those eligible units who reported participation in AFDC to the CPS and those who did not. The group of eligible reporters rather closely fits the stereotypical pattern for AFDC of the large welfare family headed by a single woman with little other income or earnings. The administrative data are close to the profile of eligible reporters on many dimensions, although they do not exhibit quite as high a proportion of cases fitting the stereotype. At the other extreme, eligible units not reporting AFDC include a much higher proportion of smaller units headed by a married couple with some other income and earnings (see Table 1).

Eligible reporters accounted for about half the total eligible population in both 1987 and 1983. However, for reasons that could include differential undercoverage and reporting errors in the CPS, they accounted for only about two-thirds of the total number of participants in each year needed to match the

administrative data.¹⁶ Hence, even if the TRIM2 AFDC participation function automatically assigned participant status to all eligible reporters, which it does not, it would still have to draw from the pool of eligible nonreporters to reach the target caseload. The calibration process in 1987 produced a simulated participating caseload that resembled the administrative data on the dimensions of unit size, age of youngest child, sex, marital status, and race of head but not on the dimensions of relationship of head to youngest child, relationship of the

unit head to the household head, gross income, and earnings. The calibration was more successful in 1983, deviating substantially from the administrative profile of the AFDC caseload only on the dimensions of relationship of the unit head to the household head and gross income (see Table 2).

The discrepancies between the March CPS profile of simulated AFDC eligible units and the administrative profile of actual participants, which make it difficult to calibrate the simulation of participation in the AFDC program, undoubtedly reflect problems with the March CPS data. However, particularly for those dimensions on which both eligible reporters and nonreporters of AFDC differ from actual participants (notably, relationship of the unit head to