nationwide sample survey of over 60,000 households conducted during selected months by the Bureau of the Census…. The CPS is designed for income analyses.” In fact, the CPS is conducted monthly and is designed to derive estimates of unemployment. Page II-4 has a footnote that states: “Statistical matching is a variation on the data collection procedures used by the Bureau of the Census and other survey research organizations called ‘hot decking.’” In fact, statistical matching is a data imputation procedure, and its relationship to hot decking is arguable. A footnote on p. II-5 implies that subfamilies are composed of single-parent daughters of family heads when, in fact, subfamilies can be intact and can be single-parent males.
These types of errors begin to erode the reader’s confidence in the model because it appears that Lewin/ICF, Inc., did not have a complete understanding of the underlying data. Passages such as the following further erode the reader’s confidence (Lewin/ICF, Inc., 1988: IV-88):
The average LIHEAP benefit amounts reported in the March 1987 CPS varied substantially across income, fuel type, household size, and Census region groups. However, due to limited sample size, some of the estimated average benefits varied more than we felt reasonable. To reduce this apparently spurious variation, we estimated a regression of reported LIHEAP benefits as a function of income, household size, fuel type, and Census region. We then solved the estimated regression model to obtain estimates of average benefits for each income/fuel type/household size/Census region group as shown in Table 38. In the HITSM simulations, eligible individuals were assigned the average benefit reported in the CPS for households of similar characteristics using the data in Table 38.
Although it is not clear what is meant by “solved the estimated regression model,” this passage seems to indicate that Lewin/ICF did not believe the reported data because of excessive variation, so Lewin/ICF estimated a model with discrete independent variables and assigned predicted means to all observations.
Suggestions
Clearly, many of the problems with the HITSM documentation may simply reflect poor technical writing and may not be indicative of the quality of the model. However, Lewin/ICF needs to understand that in microsimulation, where the particular policies or scenarios being analyzed may never occur, the audience for the projections and analyses must rely on the communicated result to formulate judgments about a model. As a technical report, the HITSM documentation would be passable after careful editing and production. As far as providing model documentation to outside users of the model, the document the panel reviewed is inadequate.
