Research, Experimentation and Evaluation for the Decennial Census
Prof. Lawrence D. Brown
Miers Busch Professor of Statistics
The Wharton School
University of Pennsylvania
Chair, Committee on the Design of the 2010 Census Program of Evaluations and Experiments
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
National Research Council
The National Academies
Subcommittee on Federal Financial Management, Government Information, Federal Services, and International Security
Committee on Homeland Security and Governmental Affairs
March 5, 2009
Good afternoon Mr. Chairman and members of the Subcommittee. I am Lawrence Brown, professor of statistics at the Wharton School of Business and a member of the National Academy of Sciences. As such, I have been actively interested in issues relating to the decennial census for over a decade, especially since my testimony before Congressional subcommittees in 1997 and 1998. 1 have written two scholarly articles about the operation of the decennial census, and-perhaps more pertinently---have served on several National Academy of Sciences/National Research Council advisory panels involving census issues. This service includes the 1998-2004 Panel to Review the 2000 Census and the 2005-2008 Panel on Coverage Measurement for the 2010 Census. I am currently the Chair of the Panel to Review the 2010 Census Program of Evaluations and Experiments. Many of my comments this afternoon are drawn from a very recent Letter Report of this Panel, mailed to Mr. Thomas Mesenbourg as Acting Director of the Census Bureau.
As you know, the decennial census is a complex, expensive operation. It has been described as the nation's largest peacetime mobilization. Research and development for the present, and looking to the future, are essential if this operation is to be completed now and subsequently in an accurate and acceptably economical manner. There are three issues I would like to bring to your attention from our panel's interim report in 2008 and our current letter report to Mr. Mesenbourg. Two of these issues involve research and planning that should be part of the 2010 Census if the census is to remain a satisfactorily contemporary operation in 2020. The third concern is a more immediate one about research that should be conducted before fielding the 2010 Census.
Before coming to the two issues of importance for the next decade, I'll first address the more immediate concern. This concern arises as a consequence of the replacement of handheld computing devices for use in the nonresponse follow-up portion of the census. This operation-acronymically termed NRFU-is the single largest and most expensive part of the census, with over half a million census workers operating out of regional and local offices throughout the country to actively collect census data. Several members of your committee, as well as other members of Congress, and others in the government have already noted the unfortunate and considerable increase in census costs arising out of the failed effort to incorporate handheld devices into NRFU. I want to focus on a different aspect of this forced change in census plans,
These devices and other equipment and software accompanying; them had been designated to form the core of the Operating Control System. This is the system used to keep coordinated track of, and direct, daily operations for the army of field workers involved in NRFU. The Census Bureau is now in the process of restructuring the entire OCS. It has been reported that the new command control system will contain portions of the original design (to be supplied by the Harris Corporation, the original contractor), portions from other contractors, and components analogous to paper-based and computer-based systems used in the 2000 census.
Given the complexity of conducting the decennial census, it has long been deemed essential to have a complete test “dress rehearsal” two years prior to the census so that flaws can be detected and corrected. However, the timing of the decision to revert from handheld computers to a paper-based NRFU process was such that the 2008 dress rehearsal did not test NRFU - a major gap in pre-census testing. Because NIUU was not tested, the dress rehearsal also provided no information or testing on the interaction of NRFU processes with the redesigned coverage follow- up operation and other component census processes.
The Census Bureau acknowledges that the dress rehearsal provided an inadequate test of the 2010 census processes. As a remedy, it has scheduled a number of small system and field tests of various components of the census QCS chain. However, given that the operational control system for the field data collection system will not be ready until the summer or fall of 2009, the Census Bureau has decided against a comprehensive test of the entire process due to the lack of time to design and carry out such a test.
The panel believes that this testing strategy puts the Census Bureau in an extremely risky position. This stance leaves the Bureau vulnerable to technical flaws in the census process that involve interactions of the many components and subsystems; it also fails to detect any potential problems in the interaction between the system and census workers and administrators under field conditions. The Census Bureau needs to perform as full and realistic an operational test as feasible of all nonresponse follow-up systems from start to finish.
It is true that ideally tests should be conducted in enough time to detect-and correct-any problems. But if time is too short to allow for a full cycle of test and correction, earlier detection of defects or inefficiencies can still be vital. Even if a flaw is discovered too late to be addressed in a pre-tested, systematic way, some contingency planning will likely be able to greatly reduce any negative consequences for the census itself.
Now let me turn to two research issues that concern the longer range future of the census. Our reports discuss two topics with strong potential effects on cost: and quality--Internet data collection and the use of administrative records. Neither of these is scheduled to play a major role in die 2010 census. But 2010 is very nearby in terms of planning and implementing a large operation like the Census. So it semis much too late to fill these omissions in 2010. What concerns us now is the lack of effective plans to use the opportunity of the 2010 Census to conduct research. Such research should study how (and whether) these modern tools should be part of the 2020 Census.
In the past two decades the Census Bureau has compiled an extensive database by matching various federal-record personal ID systems including Social Security files and IRS identity records. Limited research uses have been made of these data, including an experiment in the 2000 census that used an early version of the database as a case study comparison for two sites in Maryland and Colorado. However, administrative records such as these are not a major part of plans for the 2010 census or its program of evaluations and experiments, despite their substantial potential for both census cost reduction and quality improvements.
Administrative records could be used to dramatically reduce the cost of nonresponse follow-up and improve the quality of the resulting data collected by avoiding inaccuracies in ``last resort`` enumerations (often supplied by proxy respondents, such as neighbors or landlords) and by providing higher quality information than is currently supplied by whale-person and whole- household imputation. In addition, administrative records could be used to target the implementation of census processes. A key example is that administrative records could identify areas in which the Master Address File (MAF) is deficient, and therefore in need of an address canvass check prior to the decennial census. It is conceivable that this approach could dramatically reduce the costs of the currently 100 percent application of the address canvassing; operation. Other potential uses of administrative records are also suggested in our reports.
Although wide-scale use of administrative records to substitute for nonresponse follow-up in 2020 or thereafter would almost certainly require a change in legislation, the potential benefits of increased use of records in census processes should be studied in order to estimate the extent to which such changes would be economically and statistically desirable. Given that the use of administrative records in such a manner provides one of the few opportunities to substantially reduce census field costs in 2020, it deserves serious attention in the planned 2010 research, experiments and evaluations.
The use of the Internet for data collection in the decennial census presents important opportunities for cost reductions and improvements in data quality. These include cost savings through the reduction in the number of forms that have to be scanned or keyed for data entry, reduction in the processing of requests for mailing of foreign language questionnaires, and savings in field work as a result of more prompt receipt of individual data. Use of the Internet may also yield quality improvements through easier access to foreign language questionnaires and online editing of census responses. Additionally, failure to allow the use of online response imposes the social cost of the Census Bureau's appearing; to be out of step with modern data collection and computing environments.
We recognize that the basic steps to implement an Internet experiment in 2010 are nontrivial. However, the panel is confident that the challenges can be overcome, even within a tight time frame, as they were when the Census Bureau added a limited online response option in 2000. In addition, the Census Bureau's own experience with Internet questionnaire development in the 2000 census can be tapped in the development of privacy safeguards, as can the experience of other countries in developing security protocols for online census response (including the 2006 Canadian census internet option which involved Lockheed Martin as a major contractor).
I would like to close with a brief, general observation of my own. This observation is not based on our Panel's current report, but is certainly not contradicted by anything in our reports. The Census Bureau needs an aggressive, well coordinated, forward looking research program. Such a program requires well-trained and creative personnel with broad authority to initiate research and the expectation of being allowed to fallow 'through to recommend improvements and innovations. This research effort requires high ranking leadership that deserves and gets the attention and respect of the Director of the Census. There should thus be established a stable unit with a reasonable expectation of adequate and continuing funding throughout the decade, and this unit should be responsible for research and development of effective innovations for future decennial censuses and other Bureau surveys and programs.
Thank you for the invitation to testify today. I would be happy to address any questions the Subcommittee might have.
An archived webcast of the hearing can be found on the Senate Homeland Security and Governmental Affairs Web site.