Measuring What We Spend: Toward a New Consumer Expenditure Survey (2013)
Chapter: Appendix A: Dissent and Panel Response
This dissent is intended to clarify some of the concerns of three of the panel’s economist members. Enormous effort went into this panel report and is reflected in its many good elements, which we wholeheartedly endorse. For example, the report describes several prototypical surveys. Exploring these survey types to their limit is the right approach to determine the best form for a new survey, although we are concerned about the extent to which resources will be available to BLS to pursue this course.
Despite the efforts that went into the report, some important issues did not get the attention that we think they deserve. This result is not surprising given the short time frame and a mission for the panel that included convening a data producers’ workshop and commissioning and examining two prototype surveys from contractors, and a set of requirements that is challenging indeed (Casey, 2010; Henderson et al., 2011). In addition, some important information only became available to the panel late in the process. The purpose of this dissent is to highlight several issues and to stress information that the panel as a whole did not have the time to fully discuss. In particular:
- The report emphasizes a greater role for diary-style data collection, which we fear will lead to lower quality data. This emphasis is supported by a statement in the report that the quality of existing interview and diary survey data cannot be compared and other statements that are equivocal about their relative merits. This statement is inconsistent with recently estimated measures of bias and precision, two standard indicators of data quality that generally
In 2010, six of the eight largest categories of expenditures available in both the Interview and Diary surveys show greater underreporting in the Diary survey (two other large categories of expenditures are not collected in the Diary survey). The weighted average of Diary totals to national income account totals for categories that line up with the national accounts is 17 percentage points lower than the ratio for the Interview survey, though this partly reflects the different coverage of the two surveys.
For the 37 categories that can be compared in the Interview and Diary, the weighted average of the coefficient of variation is 58 percent higher in the Diary, with 33 of the 37 categories subject to greater variation even after accounting for differences in sample size (Bee, Meyer, and Sullivan, 2012). The higher variability of responses from the Diary survey mean that 2.5 diary responses are needed to equal the precision of one interview response, an issue that is not reflected in any of the cost accounting in the report.
Response rate differences between the surveys and the frequency of reports of no expenditures are also consistent with higher Interview survey data quality. The recent experience of Canada is instructive: a switch from interview to diary led to 9–14 percent more under-reporting on average (Dubreuil et al., 2011). This evidence is mentioned in the report, but is not deemed to have important implications for a redesign. Additional evidence of problems with diary surveys can be found in Crossley and Winter (2012). While these findings refer to existing surveys, they raise important concerns about the collection of expenditure data that relies heavily on diary methods.
The suggested redesigns, of course, aim to improve upon existing methods, although there is not a great deal of evidence to support that the effects will be dramatic. For example, monetary incentives for respondents are emphasized, but the evidence on such incentives from randomized trials suggests small effects that are even smaller for high-income households (e.g., Clark and Mack, 2009). The differential effects by income are particularly important since one of the most significant problems with the CE is lower response and under-reporting among high-income households (Sabelhaus et al., 2012).
- We believe that the discussion of the prototypes in the report should highlight the difficulty of using modeled data, data with incongruent time periods, and short panels. The charge from the BLS included the following requirement: “The CE considers the ability to support micro-level analyses over a 12-month period to be a requirement, with data collected at more than one point in time.” The report correctly advises the BLS that setting priorities is an important task, but we believe that the implications of the report’s recommendations for this requirement are not adequately addressed.
Modeling of microdata to construct data at the household level is problematic at best, especially since one of the purposes of the household data is to facilitate the design and estimation of behavioral models. While both statistical and economic theory provide a guide for modeling, even careful and well-intentioned methods may yield results with dramatically misleading policy implications such as the gross overstatement of the economic returns to obtaining a Graduate Equivalence Degree (GED) when using imputed observations in the Current Population Survey (CPS) (Bollinger and Hirsch, 2006). Along with different reporting periods for different goods, the lack of an annual record at the household level will prevent obtaining a complete picture of household spending.
A panel that lasts only a few weeks or months is too short for most changes in expenditures to occur. Even when expenditure changes do occur, the relatively higher dispersion due to collecting data over shorter intervals (e.g., two weeks) at each point in the panel may result in the dispersion of the change in expenditure being too unwieldy to be of use for standard panel data analysis. We feel that the discussion of prototypes should have emphasized the implications of the different designs for construction of datasets at the household level since, in our view, some of the designs will not produce usable data.
- More emphasis could have been given to the need to develop ways to monitor the quality of the data. Given that concerns about data quality are motivating the redesign, having a way to better monitor and hopefully improve data quality is vital. For instance, the ability to link to outside sources would be enhanced by a change in language in the introductory letter to respondents getting preapproval for such links such as is currently used in the American Community Survey (ACS), Survey of Income and Program Participation
- (SIPP), and the CPS. These links could be used to compare survey responses to administrative or other data. A second approach—the development of an intensive “gold standard” survey that could be linked to a subset of the sample—is developed in one of the prototypes. A third approach is using internal data check, such as comparisons of spending to income that would be enhanced by having spending and income for the same time period and emphasizing budget balance. Research done with the data is also an important check on the extent to which the data have sensible patterns. Some of these issues are mentioned in the report, but we believe the goals and methods of monitoring data quality merit recommendations by the panel.
- A redesigned CE could also be more informed by how survey information affects the accuracy of the CPI. Information on what categories of expenditures and expenditures by which household members appear to be under-reported could be reflected more in the designs. Ideally, the designs would reflect which categories of expenditures for which extra detail in spending would be especially helpful in reducing bias in the CPI and which other categories have price changes sufficiently correlated across different goods to indicate that current detail is more than is needed. If the structure of the CPI as a plutocratic price index is going to continue, then under-reporting by high-income households needs to be better addressed.
- Finally, the report would benefit from further exploration of economic census data and other outside sources of data to obtain detailed expenditures. Some of these sources employ crosschecks that validate the data and make it more likely to be close to the truth. These sources would be especially useful for the calculation of CPI weights and aggregate expenditure tables.
We reiterate that these comments are not meant to criticize the work of the panel, but to emphasize that it is unfinished. The CE survey is an important statistical program, and all of the panel members have their own views on what the priorities of the survey should be and how best to address these priorities. The diversity in views on this panel has been one of its greatest strengths.
Robert Gillingham
Bruce D. Meyer
Melvin Stephens
Bee, A., B.D. Meyer, and J.X. Sullivan. 2012. The Validity of Consumption Data: Are the Consumer Expenditure Interview and Diary Surveys Informative? NBER Working Paper No. 18308. Cambridge, MA: National Bureau of Economic Research. Available: http://www.nber.org/papers/w18308.pdf.
Bollinger, C.R., and B.T. Hirsch. 2006. Match bias from earnings imputation in the Current Population Survey: The case of imperfect matching. Journal of Labor Economics 24 (3):483–519.
Casey, W. 2010. CPI Requirements of CE. Available: http://www.bls.gov/cex/ovrvwcpirequirement.pdf.
Clark, S.M., and S.P. Mack. 2009. SIPP 2008 Incentive Analysis. Washington, DC: U.S. Census Bureau, Demographic Statistical Methods Division.
Crossley, T.F., and J.K. Winter. 2012. Asking Households about Expenditures: What Have We Learned? Paper prepared for the Conference on Improving the Measurement of Consumer Expenditures sponsored by Conference on Research in Income and Wealth and the National Bureau of Economic Research. Available: http://www.nber.org/chapters/c12666.pdf?new_window=1.
Dubreuil, G., J. Tremblay, J. Lynch, and M. Lemire. 2011. Redesign of the Canadian Survey of Household Spending. Presentation at the Household Survey Producers Workshop, June 1–2, Committee on National Statistics, National Research Council, Washington, DC. Available: http://www.bls.gov/cex/hhsrvywrkshp_dubreuil.pdf.
Henderson, S., B. Passero, J. Rogers, J. Ryan, and A. Safir. 2011. Consumer Expenditure Survey (CE) Data Requirements. Bureau of Labor Statistics, U.S. Department of Labor. Available: http://www.bls.gov/cex/cedatarequirements.pdf.
Sabelhaus, J., D. Johnson, S. Ash, D. Swanson, T. Garner, J. Greenlees, and S. Henderson. 2012. Is the Consumer Expenditure Survey Representative by Income. Paper prepared for the Conference on Improving the Measurement of Consumer Expenditures, sponsored by the Conference on Research in Income and Wealth and the National Bureau of Economic Research, Washington, DC, December. Available: http://www.federalreserve.gov/pubs/feds/2012/201236/201236pap.pdf.
PANEL’S RESPONSE TO DISSENT
The uses of the Consumer Expenditure Surveys (CE) are complex and diverse. Our panel reflected that diversity and complexity. The members came together from different academic disciplines and collectively provided high levels of expertise in economic analysis, survey measurement, data collection, sample design, and technology. All of these skills were critical to the panel’s work, and each area of expertise reflected insight, ideas, and opinions important to designing high-quality consumer expenditure surveys and to preparing a consensus report.
During panel discussions, three members continually emphasized the importance of the CE for economic analysis based on longitudinal microdata. Their ideas are important and, indeed, are part of the whole that is the panel’s report. These members contributed substantially to both the substance of the report and to the balance and tone in which ideas have been laid out. The report is better because of the juxtaposition of their insights
and ideas with those of other panel members, and we sincerely thank them for their efforts on this report. We regret that they have now decided to emphasize their personal perspective through a dissent.
A prominent theme of the dissent is the relative quality of the current CE Interview survey in comparison with the Diary survey. The two surveys were designed differently in order for each to collect certain types of expenditures. The panel demonstrates in this report that both surveys suffer from substantial data quality issues. BLS currently uses results from both surveys, publishing estimates from one or the other of them after considering the overall quality for each individual expense item.
The majority of the panel thinks that the three dissenting members, by implying that diary-style data collection invariably produces lower-quality data than recall-style data collection, have reached a premature conclusion. The majority concluded that it is neither possible nor necessary to assess which of these two current surveys would be “better” for collecting all types of expenditures. Instead, the panel kept its focus on how to use a combination of improved modes, along with newer collection technology, external data, and respondent incentives, to create a more effective CE survey in the future. We believe the report has done that well. It provides several design options, including a supported journal using a tablet computer that is distinctly different from the current Diary survey. Each option has a different mix of interview-like and supported journal components, all of which should receive serious consideration and evaluation.
Some of the other issues in the dissent—modeling, the length of reference periods for data collection, and linking to administrative data for a richer dataset—relate directly to the dissenters’ focus on creating the most useable microdata for analysis. Each of these issues is discussed in the report, both in a broad context and in relationship to creating microdata.
The report proposes a number of ideas to improve the understanding of expenditure data and their quality, including the evaluation of intensive methods to reconcile household income and expenditures, the use of extant administrative data for quality assessment, research on how households keep financial records, and the possibility to augment the sample through more intensive interviews with a subsample of wealthier households. All are discussed in some depth in the report.
The need for an incentive program is thoroughly developed in the panel’s report. Several panel members have considerable personal experience in using and evaluating incentive programs. Knowledge of the extensive scientific literature led to our recommendation that fairly substantial incentives be used in order to affect response behavior. The report has also thoroughly developed issues related to the use of outside sources of data to obtain detailed expenditures. The panel encourages BLS to continue to
research and evaluate these options, but we concluded that none of them is currently a feasible alternative given the associated risks and costs. Far from being incomplete, we believe our report, with its broad discussions, will be an important tool for the administrators and policy makers who are responsible for determining the next steps for the CE.
