Improving Information for Social Policy Decisions -- The Uses of Microsimulation Modeling: Volume II, Technical Papers (1991)
Chapter: Reporting Errors
imputed, and the mean value of the imputed face amount was twice that of the reported amount ($20,893 versus $9,198). Although relatively few low- to moderate-income households with vehicles had the value imputed, the imputed amounts were higher on average than the reported amounts.
Undoubtedly, the current income and asset income imputation procedures in the SIPP distort at least somewhat the picture of the population participating in programs such as food stamps. Allin and Doyle (1990) compared program participants from the 1984 SIPP panel whom they simulated to be eligible for food stamp benefits with participants whom they simulated to be ineligible because of excessive incomes or asset holdings. (Seemingly ineligible participants accounted for 11% of total participants.) They found that only 5 percent of the eligible participants but 28 percent of the ineligible participants had some or all income or asset values imputed.
Reporting Errors
In addition to errors that are inadequately compensated for or actually introduced as a result of imputing for nonresponse, there are many kinds of reporting errors—known, speculated about, and unknown—that affect data quality in the CPS and SIPP. (Reporting errors can originate from the designated respondent, a proxy respondent, or the interviewer.) Aggregate comparisons of CPS and SIPP data with estimates from independent sources can indicate the magnitude of differences although they cannot pinpoint the sources of error. Such comparisons for income indicate that the combined effect of underreporting, overreporting, and misreporting errors (for example, reporting AFDC benefits as general assistance) is almost always a net shortfall.
Tables 4 and 5 show comparisons of SIPP quarterly and CPS annual income estimates by source with independent estimates for 1983–1984. The CPS obtains higher estimates of aggregate earnings (99% of the benchmark compared with 93–95% in SIPP), but lower estimates of SSI (85 versus 90–99%), social security (92 versus 100–102%), AFDC (76 versus 76–86%), food stamps (71 versus 83–90%), and veterans’ compensation (63 versus 75–80%). CPS estimates of aggregate interest, after implementation of a new imputation procedure that makes use of matched IRS data, are currently more complete than SIPP estimates, but SIPP provides more complete estimates of aggregate dividends.10
TABLE 4 A Comparison of SIPP and CPS Estimates With Independent Estimates of Income Recipients and Amounts for Selected Income Types
|
Income Source |
SIPP as a Percentage of the Independent Estimate of Average Monthly Recipients |
SIPP as a Percentage of the Independent Estimate of Quarterly Income |
CPS as a Percentage of the Independent Estimate of Annual Income (1983) |
|
Wage and salary |
|
||
|
3rd Quarter 1983 |
N.A. |
95.0 |
99.0 |
|
4th Quarter 1983 |
N.A. |
94.3 |
|
|
1st Quarter 1984 |
N.A. |
93.2 |
|
|
2nd Quarter 1984 |
N.A. |
94.4 |
|
|
3rd Quarter 1984 |
N.A. |
95.2 |
|
|
4th Quarter 1984 |
N.A. |
94.5 |
|
|
Federal supplemental security income |
|
||
|
3rd Quarter 1983 |
92.0 |
89.8 |
84.9 |
|
4th Quarter 1983 |
91.3 |
93.5 |
|
|
1st Quarter 1984 |
94.8 |
96.4 |
|
|
2nd Quarter 1984 |
98.2 |
97.4 |
|
|
3rd Quarter 1984 |
98.3 |
98.6 |
|
|
4th Quarter 1984 |
98.1 |
99.2 |
|
|
Social security |
|
||
|
3rd Quarter 1983 |
99.2 |
99.6 |
91.7 |
|
4th Quarter 1983 |
96.3 |
100.6 |
|
|
1st Quarter 1984 |
97.3 |
100.5 |
|
|
2nd Quarter 1984 |
97.7 |
101.1 |
|
|
3rd Quarter 1984 |
97.5 |
101.3 |
|
|
4th Quarter 1984 |
97.5 |
101.6 |
|
|
Aid to Families with Dependent Childrena |
|
||
|
3rd Quarter 1983 |
78.5 |
76.2 |
76.0 |
|
4th Quarter 1983 |
79.2 |
78.5 |
|
|
1st Quarter 1984 |
84.5 |
85.3 |
|
|
2nd Quarter 1984 |
86.0 |
86.0 |
|
|
3rd Quarter 1984 |
82.0 |
80.2 |
|
|
4th Quarter 1984 |
80.7 |
78.8 |
|
|
Food stamps |
|
||
|
3rd Quarter 1983 |
89.5 |
90.1 |
71.2 |
|
4th Quarter 1983 |
91.0 |
83.1 |
|
|
1st Quarter 1984 |
90.8 |
85.2 |
|
|
2nd Quarter 1984 |
90.5 |
86.2 |
|
|
3rd Quarter 1984 |
90.3 |
84.6 |
|
|
4th Quarter 1984 |
91.7 |
83.6 |
|
|
Veterans’ compensation or pensiona |
|
||
|
3rd Quarter 1983 |
89.2 |
78.9 |
63.3 |
|
4th Quarter 1983 |
89.7 |
79.9 |
|
|
1st Quarter 1984 |
90.6 |
78.0 |
|
|
2nd Quarter 1984 |
90.8 |
74.5 |
|
|
3rd Quarter 1984 |
89.8 |
76.3 |
|
|
4th Quarter 1984 |
93.3 |
79.7 |
|
|
NOTES: N.A., not available. SIPP estimates are from the 1984 panel; CPS estimates are from the March 1984 income supplement. aRecipients exclude dependents covered by payments. SOURCE: Jabine, King, and Petroni (1990: Table 10.1). |
|||
|
|
municipal bonds. The available independent benchmarks are also subject to errors of various kinds—for example, there are economic incentives to conceal income from the IRS—while the quality of household-sector estimates of property income in the national accounts likely suffers from the fact that the estimates are calculated as the residual remaining after the income accruing to other sectors is accounted for. See Vaughan (1988) for discussion of these points. |
TABLE 5 SIPP and CPS Estimates Compared With Independent Estimates of Aggregate Interest and Dividend Income, 1983–1984 and Calendar Year 1984 (in billions of dollars)
|
|
|
1984 |
|
|
Income Source |
1983–1984 (NIPA-Based) |
NIPA-Based |
Reported to IRS |
|
Interest |
|
||
|
Independent estimate |
$239.1 |
$254.6b |
$176.4 |
|
Survey estimate |
|
||
|
SIPP |
$115.7 |
$115.4 |
|
|
CPS |
|
||
|
Original imputation |
$104.7 |
$109.2 |
|
|
Revised imputationa |
$129.5 |
$138.7 |
|
|
Survey estimate as a % of the independent estimate |
|
||
|
SIPP |
48.4 |
45.3 |
65.4 |
|
CPS |
|
||
|
Original imputation |
43.8 |
42.9 |
61.9 |
|
Revised imputation |
54.2 |
54.5 |
78.6 |
|
Dividends |
|
||
|
Independent estimate |
$63.6 |
$66.5b |
$50.6c |
|
Survey estimate |
|
||
|
SIPP |
$38.3 |
$40.3 |
|
|
CPS |
$29.1 |
$30.7 |
|
|
Survey estimate as a % of the independent estimate |
|
||
|
SIPP |
60.1 |
60.7 |
79.7 |
|
CPS |
45.8 |
46.1 |
60.6 |
|
aThe revised imputation corrects for bias in the pre-1983 imputation procedure by making use of matched IRS information on interest income. bPersonal income aggregate from the National Income and Product Accounts (NIPA) adjusted to the survey universe based on observed relationship between NIPA aggregate and independent estimate for the CPS universe in 1983. cTotal domestic and foreign dividends received. SOURCE: Jabine, King, and Petroni (1990: Table 10.3). |
|||
The new processing system just introduced for the March 1989 CPS produces somewhat higher aggregate income estimates than previously obtained (by 1% overall and 3% for unearned income), because imputations are now based on the 49 separate income types that are collected rather than on 11 combined income types (see Welniak, 1990). As the Census Bureau admits, the introduction of the new processing system, which also made other improvements in the imputation methodology, is “long overdue” (Bureau of the Census, 1990). (The expanded income questionnaire was introduced in the March 1980 CPS,
but the processing system to handle the additional detail was not implemented until last year.)
Only one recent study has analyzed reporting errors in the March CPS. Coder (1990) examined reporting of wage and salary income in the March 1986 CPS for married couples who were successfully matched with IRS records.11 He found (pp. 32–33):
First, nonsampling error is much larger than that evidenced by the aggregate measures that are generally used to characterize it. The 2–3 percent underestimate in the aggregate masks widespread over- and underreporting of amounts and imputation procedures that account for a small proportion of the bias introduced by nonresponse. Second, the most serious error problems seem more concentrated at the bottom and top of the distribution. Third, in spite of these errors, the distribution of wage and salary income from the CPS for the matched universe was very similar to that derived from the IRS data…. This does not say that the estimates of the characteristics of persons at specified points in the distribution, notably the tails, are not biased in some way by these errors. Estimates for one universe in particular, the working poor, may be especially at risk.
A number of studies have investigated some types of errors in SIPP. One important study involves checking reported amounts for various transfer payments against administrative records of amounts actually paid out. The SIPP record check study compared data from the first two waves of the 1984 panel with administrative data from four states for eight programs—AFDC, SSI, food stamps, social security, veterans’ compensation and pensions, federal civil service retirement, unemployment insurance benefits, and workers’ compensation (Marquis, Moore, and Huggins, 1990). The results show negatively biased participation rates for most programs: that is, net underreporting of participation, although there are overreports as well as underreports. For most programs, there appears to be relatively little bias in reporting of benefit amounts for those
