5

Evaluation of the Demonstration Data on Housing and Tenure

HOUSEHOLD COMPOSITION

Rachel Cortes (Claritas) with co-author Ken Hodges presented a paper in “Use Cases Part II.” She stated that Claritas is a private company that builds nationwide demographic estimates and projections at the block group level, and it relies on Census Bureau data from the decennial census and the American Community Survey to choose inputs and benchmarks when it creates its estimates. Claritas began evaluating these projects beginning with the first release of demonstration data and the 2019 workshop. The most troubling feature Claritas and others found was inconsistency between population and household files. Cortes offered examples such as “improbable and impossible” values for household size, as well as areas with households but no population. For the Public Law 94-171 file, they “made peace with the fact there will be some improbable values.”

Cortes explained that the most important input that Claritas uses from the decennial census data is the cross-tabulation of age and sex. When comparing the Summary File 1 (SF1) with differential privacy products, she stated that she and Hodges calculated an index of dissimilarity that has shown improvement since 2019: the mean index of dissimilarity in 2019 was 35.4, and it is now 8.9.

Another important component from the Demographic and Housing Characteristics (DHC) file is householders by age, which enables a calculation of household income by age that can be used for marketing research. Claritas calculated the component of householders by age for nine age categories. Table 5-1 shows the block group mean percent of the population

TABLE 5-1 2010 DHC Demonstration Data vs. 2010 SF1: Block Group Mean Percent of Population That Is a Householder

NOTE: DHC = Demographic and Household Characteristics File; DP = differential privacy; SF1 = Summary File 1.

SOURCE: Adapted from Rachel Cortes workshop presentation, June 21, 2022.

of householders. In an analysis of percent of householders from the differentially privatized file compared with that of the SF1 file, the former shows higher householder categories, especially for older age groups. For example, 67.3 percent of individuals 85 and older were householders in the SF1 file, compared with 87.7 percent in the differentially privatized demonstration data.

When looking at block groups, Cortes discussed assessing the difference between the ratio of the householder rate in the differentially privatized data and SF1. These rates were calculated from the differentially privatized demonstration data from those measured in the SF1 data and taking the absolute mean difference, so negative and positive differences would not offset each other. As Table 5-2 shows, the mean difference for each age category increases as age increases.

Cortes also discussed initial analysis for householders by race and ethnicity for 14 categories (seven races by Hispanic or non-Hispanic). Although this analysis is still ongoing, she stated that early findings suggest that these data look worse than age of householder. Table 5-3 shows that the SF1 data had no inconsistencies at the block group level, while 60.7 percent of block groups had at least one race/ethnicity with more householders than the population in that group.

Cortes concluded by noting that DHC variables that combine population and householders are important to their work. While Claritas looks forward to receiving the final product, it is bracing for what it will find and what it will need to do to deal with these data.

TABLE 5-2 Mean Absolute Difference for DP Ratio vs. SF1 Ratio

NOTE: DP = differential privacy; SF1 = Summary File 1.

SOURCE: Rachel Cortes workshop presentation, June 21, 2022.

TABLE 5-3 Race and Hispanic Ethnicity: Percent of Block Groups with Householder/Population Rates Greater Than 100

NOTE: DP = differential privacy; SF1 = Summary File 1.

SOURCE: Adapted from Rachel Cortes workshop presentation, June 21, 2022.

HOUSING TENURE

Leslie Reynolds (Cornell University), with co-author Jan Vink, introduced her presentation, noting that she would be discussing types of housing tenure and how producing these data differentially impacted their accuracy. She stated that homeownership disparities are known: one is more likely to own a home if they are White, married, or generally privileged in society. Consequently, Reynolds emphasized that having inaccurate data in these situations would likely produce disparities and further inequities. Because natural clustering occurs among people who either rent or own their homes, Reynolds asserted that these areas do not necessarily fit within the TopDown Algorithm (TDA) of differential privacy.

Three research questions guided her analysis:

1. Does the TDA impact the comparisons between aggregate groups (e.g., state, county, and national levels)?
2. Does the TDA differentially impact accuracy within tenure groups?
3. Does aggregation level matter for the accuracy of the tenure areas?

Reynolds stated that she used 2010 SF1 and 2010 DHC demonstration data retrieved from IPUMS. The analysis excluded Puerto Rico and tracts with fewer than 200 households. In addition to all tracts in the United States, all tracts in New York State, Monroe County (Rochester), and Onondaga County (Syracuse) were analyzed. The analytical variable was dominant tenure tract: Owned-majority area was defined as greater than or equal to 80 percent owned households in a tract. Rental-majority area was defined as less than or equal to 20 percent households owned. Reynolds stated that the metrics of interest were bias (mean error), accuracy (mean algebraic percent error [MALPE] and median absolute percent error [MdAPE]), and precision or usability. Precision was defined as having a mean error and MdAPE greater than or equal to 10.

Reynolds noted that without separation by tenure count discrepancies between the files are slight for housing units. Large households had the highest MALPE: five percent (see Table 5-4). Reynolds stated that these areas are most likely to include members of underserved populations, such as people of color, younger people, and those with lower incomes.

The degree of error was larger in the rental-majority areas versus owned-majority or mixed areas. Reynolds stated that the maximum degree of error was 11, found in Onondaga County for households with children in rented-majority areas (see Table 5-5). When focusing on households with children, Figure 5-1 shows that the MdAPE was largest for households in rented-majority areas and smallest in owned-majority areas. The minimum MdAPE was found in owned-majority areas of Monroe County (1.4%).

TABLE 5-4 Differences in Mean Counts, Mean Error, and Mean Percent Error Between 2010 SF1 and Demonstration DHC File

NOTE: DHC = Demographic and Housing Characteristics File; MALPE = mean algebraic percent error; SF1 = Summary File 1.

SOURCE: Leslie Reynolds and Jan Vink workshop presentation, June 22, 2022.

TABLE 5-5 Mean Error by Households with Children, Geographic Area, and Tenure Majority

*Errors significantly different from zero.

SOURCE: Leslie Reynolds and Jan Vink workshop presentation, June 22, 2022.

Rental-majority areas had the most noticeable accuracy issues across all aggregate levels. The maximum MdAPE was found in rental-majority areas of Onondaga County (14.2%). Reynolds stated that the differences could be due to the privacy-loss budget and unintended allocations, which may change when the TDA is finalized, but disparities such as those shown for rental-majority areas are still problematic.

Even greater disparities are demonstrated in Figure 5-2 for inaccuracies reported for large households (defined as having five or more people). Owned-majority areas had the lowest MdAPE, with a minimum of four percent for owned-majority areas in New York State and Monroe County. Rental-majority areas in each aggregate level had the highest MdAPE, with the exception of mixed-tenure areas of New York State, which had an MdAPE of 6.4 percent. The maximum MdAPE was 21.9 percent for rental-majority areas of Onondaga County. When looking at precision across all tracts in the United States, the rental-majority areas were least precise and owner-majority areas were the most precise, as shown in Table 5-6.

Reynolds offered several conclusions. She cautioned that broad measures of similarity between SF1 and the demonstration data can be misleading. In her analysis, she found that tracts aggregated to substate levels generally produce the largest errors. When examining the metrics by tenure

TABLE 5-6 Share of Tracts in the United States with Large Errors

NOTE: HH = household.

SOURCE: Leslie Reynolds and Jan Vink workshop presentation, June 22, 2022.

majority, accuracy differences were revealed. Data on households in rental-majority areas tended to least accurately match the original 2010 SF1 data, and data on households in owned-majority areas most accurately matched the SF1. Reynolds also stated that measurement of households with children and large households in rental-majority areas contained the most bias and also had the lowest levels of precision. However, the issues of bias and accuracy were less prominent in the person files.

This page intentionally left blank.