things they represent. These challenges place inherent limitations on what we can infer from any given source and the confidence we can have about our inferences. At the same time, though, throwing our hands up in resignation is not only counterproductive; it also fails to recognize that these flawed sources of information constitute some of the best knowledge we have about our social and political world.

One of the principal approaches for understanding the values and constraints of various data streams (or data sources) and expanding the limits of our knowledge comes through data integration. Data integration is the process of taking multiple diverse streams of information and finding ways to reconcile their differences, use them jointly to make conclusions, and/or determine where they provide complementary or conflicting understandings. There are many related terms describing aspects of this process: data aggregation, data combining, data blending, data pooling, data triangulation, data linkage, data merging, and more. In this chapter, we use the term “data integration” to denote the larger set of procedures that encompasses all terms (hence, the bolded definition above). Integrating across different data sources can provide critical information about the quality of the data we have gathered, the confidence we should have in a particular conclusion, and the mechanisms through which some process is operating, and sometimes data integration is necessary to measure the very phenomena we wish to study.

Integrating data takes a variety of forms, as shown in Figure 3D-1, often depending on what types of data are available as well as key aspects

of the data. Differences in the source, scope, and granularity of data change the affordances of both individual datasets and data integrations for analysts. We discuss a series of key considerations in Section 2 related to understanding different types of data and how they relate to research objectives.

The benefits of successfully integrating data can be enormous for the intelligence community (IC) analyst. Evidence from connecting different kinds of data about the same phenomenon can allow analysts to examine whether their conclusions are robust to alternate forms of measurement (Section 3.5). When the results from a social media analysis echo those of a traditional sample survey, an analyst can have greater confidence that the findings were not a fluke arising from a single idiosyncratic dataset. Furthermore, datasets may be available spanning different time points. The integration of such datasets is often necessary to understand how things have changed in a particular location (Section 4.3). These sorts of data integrations may allow analysts to discern trends, which themselves might answer key questions such as whether and how protest events shape approval of political leaders. Integrations involving data streams collected from different communities or places are central to many comparative analyses, and they allow analysts to identify differences in attitudes and behaviors across locations that may flag emerging issues or the need for interventions (Section 4.2). Integrating multiple sources of data that contain information on particular individuals or organizations can also provide a clearer picture of key phenomena of interest (Section 4.1). Understanding how individual attributes relate to network characteristics can help explain why networks are structured the way they are and reveal important patterns, providing evidence, for instance, of how knowledge was shared among the 9/11 hijackers (Krebs, 2002). All these conclusions depend on connecting multiple sources of data.

When analysts integrate different data streams together, a variety of questions emerge about how the data should be understood, what factors need to be considered, and how we can use the full information from available data to reach the best possible conclusions. Answering these questions depends on the research goals as well as on the characteristics of the available data (for an example, see Schober et al., 2016). In general, attempts to integrate different sources of information begin with an assessment of the strengths and weaknesses of the individual datasets that are being used (see “Drawing Inferences from Public Opinion Surveys: Insights for Intelligence Reports” [“Drawing Inferences”] earlier in this Analytic Framework for examples). Determining what sources of error might exist within a given stream (Section 2.6) is critical for understanding the utility of the data as well as considering how those errors can be mitigated. At the same time, it is important to consider the requirements for a given project. Some intelligence projects need to achieve high levels of precision, must be accurate

on the individual level, or must thoroughly capture the distribution of attitudes or behaviors in society. Other goals are less sensitive to these sorts of considerations, but may need to carefully determine whether a proposed relationship is causal or mere coincidence. Whether a given method is “fit for purpose” (Baker et al., 2010) depends on how data quality, research questions, and inferential methods relate. Small errors in sample composition, for instance, can undermine conclusions about who is likely to win a close election but may be inconsequential when trying to understand how individuals in different groups are going to vote.

In the rest of this chapter, we outline some of the key considerations to think about when combining different data streams into a single analysis (Section 2), and discuss a number of technical procedures relevant to improving the quality of the data that are used (Section 3) and the inferences that are made from those data (Section 4).

SECTION 2: KEY CONSIDERATIONS FOR DATA INTEGRATION

Different types of data can be connected to one another and combined in a variety of ways as illustrated in Figure 3D-1. Decisions about what kinds of data to combine and to what ends raise a wide scope of possible considerations for data integration. In general, the factors that researchers need to think about or potentially worry about depend on what population(s) analysts hope to make conclusions about; the types of conclusions they aim to make; the reason they are conducting their analyses; the granularity of the available data; the similarities and differences between the methods of the different data streams that are being combined; and the temporal, social, political, and linguistic contexts across which data were collected. Accounting for these facets of the data allows analysts to see what kinds of insights each data stream can enable as well as recognize sources of variability in the answers that might be obtained. In turn, this helps highlight where the results of different streams may yield similar insights and where aspects of data collection methodology could induce differences. It also implies circumstances where various types of analytic approaches and data adjustment procedures may be warranted (discussed in Section 3).

Consider a case where an analyst is interested in comparing global opinions toward the United States from populations in several different countries. Multiple data collection methods might serve this goal, each with their own strengths and weaknesses (as illustrated in “Alternatives to Probability-Based Surveys Representative of the General Population for Measuring Attitudes” [“Alternatives to Probability-Based Surveys”] earlier in this Analytic Framework). One approach could rely on publicly available datasets such as those produced as part of the Pew Research Center’s Global Indicators, while another might involve designing and implementing a new

transnational survey, another alternate strategy could focus on extracting sentiment from social media posts about the United States, and yet another method might involve relying on country experts. Each of these strategies has a place in the analyst’s canon, though which approach is preferable will likely depend on a variety of factors and considerations (e.g., urgency, available financial resources, countries of interest, accessibility, and infrastructure in the target countries). Collecting novel survey data requires considerable lead time and large financial resources. Social media analytics may yield data much more quickly but may be distorted if large segments of a society lack sufficient internet access. Similar considerations exist across all types of data sources (see Table 3D-1). In all situations, how well the analyst is informed about the five components in Figure 3D-1 (data sources, data scope, data granularity and continuity, social climate, and political climate) in relation to the analytic goal is likely to determine the ability to meet this goal.

This section outlines a series of key considerations for thinking about how data can and should be combined. Throughout this section, we refer to Table 3D-1, which illustrates some of the challenges associated with various types of data that might be integrated, and Table 3D-2, which lays out a series of sources of inconsistency that can introduce errors into integrated analyses.

As the tables illustrate, myriad factors lead different data sources to yield somewhat different conclusions. This variation is to be expected, as social measurement is inherently somewhat imprecise. Integrating datasets may initially seem to only highlight the inconsistency or variability of estimates, but when we understand the sources of differences, we can account for them and generate converging estimates across approaches, which should induce confidence. Indeed, converging evidence that emerges without common sources of bias represents some of the most actionable social research (Campbell and Fiske, 1959). Hence, attention to biases that are likely to be present across the sources of data that are being integrated is critical for determining how confident we should be in the conclusions of analyses generated using those sources. By considering how various sources of data shown in Table 3D-1 relate to possible sources of variability shown in Table 3D-2, analysts can assess what level of convergence they should expect in their results.

2.1 Target Populations

One of the first steps for all human research involves determining the population of interest, that is, the group of people (or entities of any type, e.g., businesses) that an analyst would like to know something about and extrapolate the results to. Depending on the analytic goals of a project, this could consist of anything ranging from a small set of individuals in a critical network to entire populations at the national or regional level. As discussed

that analysts may be interested in estimating: (1) the proportion of a population that can be characterized in a particular way (point estimates), (2) changes in population attributes over time (trends), (3) the extent to which different population attributes relate to one another (relationships between characteristics), (4) the extent to which the relationship can be attributed to a causal link (causality), and (5) whether particular types of interventions can be applied to influence an attribute in a population (interventions). Going back to the example of the global assessment of opinions toward the United States, one may be interested in what proportion of the population in a particular country has a favorable opinion, how these fluctuate around the year a new U.S. president comes into office, whether the opinion differs between countries where the U.S. military is stationed versus not, whether changes in the U.S. foreign policy are lowering favorability toward the United States, and whether a media campaign may improve the pro-U.S. sentiment. These different goals alter the data that are needed to reach a particular conclusion, the set of potential factors that should be analytically addressed, and the sensitivity of the conclusions to violations of various assumptions.

Point Estimates

Point estimates provide a snapshot about a population such as the proportion of people who support a particular leader at a particular moment. For point estimates, it is important that the data accurately reflect the full population. Although it is often possible to derive population point estimates from a single data collection, integrating data across sources can be valuable for improving the accuracy of estimates (e.g., by allowing for adjustments to account for coverage errors, nonresponse, or population changes; see Table 3D-2). Any given survey assessing, for example, the prevalence of a particular immigrant group in a particular country might contend with variability that is due not only to sampling error but also to data collection methods, estimation strategies, sampling strategies, assumptions about appropriate data adjustments, and idiosyncratic sources of error (e.g., social climate toward immigrants). By either producing multiple estimates or combining data from different methods to generate a pooled estimate, analysts can do better. Different approaches to estimating the same parameters are likely to have differing errors, and because these errors are unlikely to yield identical biases, when aggregated they tend to reduce the expected error in the overall estimate. The reduction in error is expected because the strength and the weakness of each data stream are addressed in aggregation methods, not because errors from different data streams are merely cancelled out through the aggregation. It should be noted that disparate point estimates are indicative of difficulties of measuring the desired construct, in that there is no single way to operationalize such a

measurement. When a new construct is measured or when measuring a well-established construct in a new setting (e.g., measuring political efficacy in North Korea), disparate estimates may suggest measurement difficulties and aggregating them may not necessarily reduce the error.

Trends Over Time

Another common reason for integrating datasets involves assessing changes over time, such as whether support for a particular military operation is growing or shrinking. In general, trends can be identified with a few different strategies depending on analytic goals. Repeated cross-sectional studies of target populations can highlight points in time when attitudes and behaviors change and how that change is proceeding (Lebo and Weber, 2015). Cohort studies, which track specific groups of individuals over time, can identify whether observed changes are due to changes in the composition of a population or due to individuals changing their attitudes and behaviors (Samet and Muñoz, 1998). Panel studies, which follow specific individuals across multiple time points, can help identify the attributes of those individuals whose attitudes or behaviors change over time (Hsiao, 2014). All analyses carried out over time can be thought of as integrations of distinct data collections from different time points, requiring researchers to think about the comparability of the data collected across time points. Integrations across time must contend with things like changing norms that influence what people post on social media sites and how they refer to things (e.g., the term COVID-19 was not mentioned during the first month of the pandemic, because the name had not yet been designated) as well as changes in the underlying population.

Relationships Between Variables

In many cases, the central question for an analyst involves how things differ across different groups of people. When an analyst wants to compare people from different countries or people in one ethnic group with those in another group, combining datasets can be important for either generating inferences or testing the quality of those inferences. If no single study covers both areas adequately, then the compilation of multiple datasets constitutes a necessary step for generating a conclusion. Here, key questions revolve around the compatibility of the different data collections, as observed differences between regions may instead reflect differences relating to the respondents from whom data was collected, how that data was collected and processed, or the political, social, and linguistic contexts that may have led to these differences. In contrast, when relations between variables are measured using multiple datasets that cover the areas in question, the

benefits are more akin to those for point estimates: increasing precision and reducing sources of variance that may be unique to one dataset or another.

Causal Relationships

Analysts frequently want to know whether something that was done had a particular effect, such as whether a new program of the U.S. Agency for International Development improved attitudes toward the United States. Demonstrating that two factors are related to one another may sometimes mislead or tempt analysts to conclude that one factor affects the other, implying a causal link. Perhaps no statistical rule has received more attention than the truism that “correlation does not imply causation.” Understanding whether a particular relation exists with or without a causal link and identifying what can be done in one factor to effect changes in the other factor requires that analysts get beyond mere numeric associations. While challenging, causal inferences are likely to be among important analytic goals of the intelligence community (especially as they relate to questions like “What intervention strategies are effective in shifting Country A’s public opinion more favorably toward the United States?” and “Was Policy B successful?”). Formally, causal relationships can be identified with confidence only where the cause precedes the effect, the cause is related to the effect, and no other alternatives explain the effect (Mill, 1898). In most contexts, causal identification is only present where experimental manipulations are introduced either by a researcher or through some other randomized or quasi-randomized process (e.g. a policy that is rolled out to a random subset of housing voucher applicants).

Assessing the likelihood of a causal relationship where randomization is infeasible or otherwise has not been implemented can often be aided by a strong theoretical understanding of how such processes typically operate, effective elimination of other alternative explanations, and by a convergence of evidence (Shadish, Cook, and Campbell, 2002). Evidence that an association is stronger in regions of a country that share a particular history, for example, may help narrow the range of possible causes for that relation, and this sort of evidence both better explains the mechanism and rules out likely alternatives. A variety of sophisticated statistical techniques used in data analysis can also help isolate likely causal mechanisms when analyzing quasi-randomized experiments or observational data:

• Mediation and moderation analyses can reveal the extent to which proposed mechanisms explain how a relationship between variables actually operates (MacKinnon, Fairchild, and Fritz, 2007)
• The use of control variables and statistical matching approaches can help rule out alternative explanations (Rubin, 1973)

• Instrumental variable approaches can test the direction of a proposed causal relation (Angrist, Imbens, and Rubin, 1996)
• Regression discontinuity designs can help link changes in an outcome with particular events (Thistlethwaite and Campbell,1960)
• Marginal structure models can correct for time-dependent confounders (Robins, Hernan, and Brumback, 2000)

These strategies are discussed in section 4.3. All these approaches address specific confounder issues in causal inference and require certain data set-ups to improve conclusion robustness. Similar data collection methods yield more robust conclusions than differing ones and data collected from similar individuals (or even better, individuals who are part of a panel) is more causally robust than data collected from distinct segments of the public.

Interventions

An analyst’s goal may not be merely to understand the population but to directly influence it, such as when deciding how to combat a disinformation campaign. Here, the analysis question extends from seeking some understanding about whether one thing causes another to determining what kinds of initiatives successfully change a particular measure of interest in a targeted way. In the context of elections, there may be interest in shifting voter preferences, or when disinformation is widespread, the goal may be to encourage people to correct problematic misperceptions. Datasets may need to be merged to ascertain how the population targeted by a particular intervention relates to the population that was not targeted by that intervention or to allow researchers to match information on the targets of an intervention to data that might reveal evidence of efficacy.

2.3 The Focus of Measurement

Another critical aspect of data integration stems from considering exactly what it is we want to measure. This is often trickier than it may initially appear. Consider a situation where a large number of people are expected to attend a political protest on some issue. Analysts may want to know a variety of very different things related to such an event. In some cases, it may be valuable to reach conclusions about the profile of particular individuals who are likely to participate in the event. In other cases, it may be useful to understand the distributions of different types of people who are at the event. In yet other cases, the relevant parameter may be the number of people who can be expected to show up for the event. The differences between these goals can be important. Aggregations of the best

estimates of individual behavior can lead to biased assessments of collective behavior if the errors in individual estimates are not carefully mitigated through any models.

There are many additional research goals, of course. In an environment where we care about messaging, we might be interested in the factors that encourage a particular behavior or instead in what novel messages might encourage people to engage in it. These can differ if the effects of messages are cumulative or if the impact of one type of message has been exhausted. With respect to the popular support of a particular policy proposed, there are again multiple distinct ways of thinking about the question, and each requires different measurement approaches. The proportion of people that support a policy is often different from the proportion that would support it if they knew more about it, for instance. The former may best capture a contemporary political environment but may also distort if there are widespread misunderstandings that could be corrected. The latter can sometimes be estimated using statistical simulations (Althaus, 1998; Bartels, 1996), provision of information, or other techniques such as deliberative polling (Fishkin and Luskin, 2005).

Finally, the data we can gather about individuals might consist of various components of beliefs, attitudes, experiences, and behaviors. Different sources of data provide different windows into these components. Expressed attitudes on Twitter, for instance, are not the same as what people say when asked a question in a survey (see e.g., Pasek et al., 2020). For experiences, behaviors, and some factual beliefs, ancillary sources of data (e.g., official records or geographic tracking) can allow for validation of a survey measure. In other circumstances, there is little option but to take a measure at face value.

The links between the different kinds of things we want to measure and the data that are available can have myriad influences on conclusions that can be drawn. While it is impossible to be generally prescriptive about how data collection should work for a particular purpose, thinking about what is being measured, the quality with which it is being measured, potential biases in that measurement, and how these all relate to the research question is critical.

Measurement error models may prove helpful. In the language of psychometrics, criterion relevance, where the actual criterion of a measured construct reflects the conceptual criterion (Messick, 1989), may not be realized if the actual criterion is deficient in capturing the conceptual criterion and/or contaminated with something other than the conceptual criterion (for example, using smartphone data to assess crowd sizes may yield biased comparisons if some groups of people in a country are more likely to have smartphones than others). Unfortunately, measurement error models often focus on avoiding contamination but not on ensuring that measures fully

capture a concept (i.e., that there are no deficiencies). Approaches to considering measurement validity can be found in section 2.7.

2.4 Data Granularity: Toplines, Crosstabs, and Microdata

Before multiple data sources can be linked, it is important to take stock of the data that are available (or are being collected) to assess what can be done with them to answer the central questions of an inquiry. For example, as discussed in the “Drawing Inferences” paper, survey data can be used to answer all sorts of questions about a particular country or population. When researchers are designing a survey and collecting the data, the conclusions that can be reached depend on the limits of the data collection itself. In contrast, when currently available data are being used, analysts are limited by the forms those data can take. Sometimes the full original dataset is available and researchers can use this for their analyses. In this case, we say that the researcher has access to the microdata (microdata from a survey may have 1,000 rows, each reflecting the answers of a single individual to a set of survey questions). In other circumstances, however, only aggregates of the data are accessible. These typically come in two forms: topline results report on the overall proportion of a population that shares a certain characteristic (e.g., an analyst looking at topline reports may only know that in October, a country’s president had a 46 percent approval rate); crosstabs reveal how these proportions vary across particular population subgroups (e.g., approval was 35 percent in one ethnic group and 53 percent in another). For these latter types of data, researchers need to be aware that many components of the data collection and analytic process are baked in (i.e., their implications need to be considered post hoc). There may be no way for an analyst to exclude an irrelevant population (e.g., non-citizens) or to make different adjustments to the data. Approaches to integrating data differ depending on this question of data granularity.

When linking microlevel datasets, conclusions can be generated for individual units. This may not be the case when higher-level aggregates of the data are being used. Despite the inherent limitations that crosstabs and toplines have, they can nonetheless still be useful. For instance, the proportion of a population that shares some belief or that is planning to vote a particular way may be a primary outcome. Topline estimates of these quantities sometimes even include adjustments to better represent the population (e.g., using survey weights). Integrating this sort of data can allow analysts to track trends over time or to consider the similarities of results across data collection methods but cannot yield individual-level insights. It also encourages different analytical approaches (see Section 4).

identical questions may yield different results depending on their locations within a questionnaire (e.g., opinion toward allowing foreign communist newspaper reporters to report from the United States is more favorable when this question is asked after than before whether other countries, like Russia, should allow U.S. reporters; [Schuman and Presser, 1996]). Seemingly slight changes in question wording can distort trends, as happened when the Current Population Survey introduced a new wording for questions about health insurance in 2014 (Pascale, Boudreaux, and King, 2016). This change in wording improved the data in that the health insurance coverage estimates were in line with similar surveys (Medalia, O’Hara, and Smith, 2016), but the change also necessitated cautioning users who tracked characteristics over time (Hill et al., 2019). Other sources of variability can include the mode of a study (e.g., online versus face-to-face), the processes people are using to provide data (e.g., speaking with an interviewer or posting a message accessible to the world), what languages are made available for participation, or even the quality of official recordkeeping (which can vary across institutions within a single country; e.g., at hospitals or morgues), and the affordances of whatever medium they are using to communicate (e.g., when Twitter changed its character limit, different messages were possible). Even when these sorts of factors look comparable, there may be important differences (e.g., survey respondents might refuse to answer a question they think comes from the government even if the question is identical to one asked by a private firm; or expressed attitudes toward racial groups may depend on the presumed race of the interviewer [Schuman and Converse, 1971, Davis, 1997]).

Nonmethodological factors can also shape the data generating process. Different survey firms may be involved in a study using a single method (e.g., the European Social Survey, which employs different firms to collect data in different regions). Depending on firms’ capacity and capability to adhere to the specified method, the results may vary. Assessment of the results also depends on the transparency of methods and the thoroughness of the information that firms provide. When integrating data across social/political contexts, a question sensitive in one context may not necessarily be so in another context. In some countries, it has been common for third parties to be present during interviews, which may alter the responses people are willing to give (Diop, Le, and Traugott, 2015; Mneimneh et al., 2015). With societal and political changes, these contextual factors may also change over time. For example, one firm that was willing to field questions about the desire for political independence in Hong Kong in 2018 was no longer willing to do so in 2020. Data from an authoritarian society may suffer from survivorship biases, where only certain results are selectively made available. When such a scenario is likely, one may consider exploring such biases through qualitative research (e.g., interviewing key experts on the topic) or available data that may allow modeling of these data hand-picking processes.

ity (e.g., Lee et al., 2019). For those who speak multiple languages, the language in which the data is collected brings cultural norms associated with that particular language to the interactions, such as survey interviews (Marian and Kaushanskaya, 2004; Ross, Xun, and Wilson, 2002), affecting measurement (Lee, Nguyen, and Tsui, 2011; Lee and Schwarz, 2014; Lee, Schwarz, and Goldstein, 2014). Linguistic differences in meaning and usage are also often present across dialects of the same language (Renschler and Kleiner, 2013).

Countries are frequently implicitly treated as a cultural unit, but this is not a correct approach. Two neighboring countries may share a similar cultural background, while a single country may be home to multiple racial and ethnic groups that do not share a unified cultural background or a single language (e.g., U.S. vs. U.S. Latinos vs. U.S. English-speaking Latinos). In fact, culture is knowledge learned and shared among a group of people through verbal and nonverbal communication (Goodenough, 1971; Hong, 2009). It is conceivable to designate comparison groups that share common experiences and knowledge rather than countries as “cultural” groups.

When analyzing data from social media, norms of site use can differ across populations. Individuals who participate in “Black Twitter,” for example, are less likely to use their real names or provide precise locations than individuals who are in other groups (Freelon, 2018). This can make it harder to generate all types of inferences about them (such as their gender) or render inferences less accurate. Name identification tools also may be more accurate for some populations than others, introducing systematic errors into analyses.

2.7.2 Measurement Equivalence

The link between cognitive psychology and survey measurement is well established (Schwarz, 2007) in that the processes of how people understand survey questions, gather relevant information, and form responses provide a framework to improve survey questions. With culture being an elementary construct of cognition, it provides a system to organize personal experiences (Nisbett et al., 2001). To illustrate: something as fundamental as time orientation, the temporal dimension of past, present, and future to which individuals assign coherent meanings to their personal experiences (Kluckhohn and Strodtbeck, 1961), covaries with culture (González and Zimbardo, 1985; Levine, 1997) and affects survey measurement (Lee, Liu, and Hu, 2017). What is required in comparative research data is measurement equivalence (Scheuch, 1968). This goes beyond surface-level considerations about cross-sections with culture (language and religion) and considers the comparability of the ways in which a given measure functions across cultures and contexts. Self-rated health, for example, is a simple survey ques-

tion proven to predict subsequent mortality in many Anglo cultures (Idler and Benyamini, 1997). Whether this predictive power holds for non-Anglo cultures is the key to the functional equivalence of the measure (McGee et al., 1999). This is a vital but difficult task that persistently burdens cross-cultural research. While measurement equivalence is an issue applicable for nonsurvey data, there is little room for researcher control in those sources, making its assessment difficult. Sometimes, known sources of inconsistency can be identified and accounted for, but at other times, sufficient information is simply not available.

2.7.3 Response Style and Reporting Heterogeneity

Even when the substantive meaning of a question is consistent across comparison groups, there can still be variations in how respondents are classified based on the way individual responses are assessed. In opinion and attitude research that analyzes subjective measures, Likert response scales that ask respondents to express their opinion and attitude on the scale of agreement (strongly agree, agree, neither, disagree, strongly disagree) are an indispensable tool particularly in public opinion research. While extremely popular, these scales are subject to biases associated with response style, a tendency to choose certain response points regardless of the question content (Van Vaerenbergh and Thomas, 2013). Response style is systematic and can be culturally distinct. Latinos in the United States, for example, are shown to acquiesce more than non-Latinos (Davis et al., 2019). Asians tend to choose middle response points more than non-Asians (Chen, Lee, and Stevenson, 1995). When comparing Latinos and non-Latinos on their life satisfaction measured with a Likert scale, it becomes difficult to ascertain whether the difference (or lack thereof) between the two groups comes from true life satisfaction or response style (Lee et al., 2020). Particularly for multi-item measures, reliability and validity may become erroneously inflated or deflated due to response style effects. There is some attempt to use item-specific response scales rather than Likert scales (Fowler, 1995; Saris et al., 2010), but the literature does not agree on its effectiveness (Liu, Lee, and Conrad, 2015; Ross, Steward, and Sinacore, 1995; Saris et al., 2010).

Reporting heterogeneity is a more general term describing variability in how individuals express their own states (Bago d’Uva et al., 2008). Response style is but one form of reporting heterogeneity. Another occurs when individuals are tempted to select verbally labeled scale points in an otherwise numerically labeled scale (Pasek and Krosnick, 2010) or when some individuals tend to select generally systematically higher values on certain scales (Wilcox, Sigelman, and Cook, 1989). When reporting heterogeneity arises at random, it is of little concern because it only introduces

noise into measurements. When it differs systematically, particularly across comparison groups, this heterogeneity confounds comparisons.

2.8 Political and Ethical Considerations

It is important to consider the implications of assembling the data that will be used for a project in all studies, but especially when combining datasets. These implications can influence the work that is being done and the potential costs and benefits associated with doing that work. These constitute important limits of data collection, either because the data desired cannot be collected, should not be collected, should not be integrated, or must be properly safeguarded if it is to be used.

One type of potential concern regards the feasibility of a particular data assemblage. Sometimes the work necessary to collect the data incurs prohibitive costs in money or time that negate the value of the data. Another involves the possibility that there will be political risks to individuals whose data is being used in a study. Some individuals may withhold attitudes that they think are politically risky or will get them in trouble if they do express those attitudes. Expectations of privacy and the risks of leaked information may differ across political contexts. There can also be social risks to individuals if the information they provide could be viewed negatively by others, leading them to provide information selectively or to experience harm if they are forthright. One notable context for social risk comes if others are present during the data collection process, which is often more common outside the U.S. context (Mneimneh et al., 2015).

Data collection and integration can also pose risks for data collectors and sponsors. Asking some questions in some contexts may be illegal, incur reputational risks, or elicit unwanted political oversight. Finally, it is important to note that there can be risks from data inferences that can be made by combining datasets that may not be present if data are not combined (Singh et al., 2020). One notable case of this occurred when a teenager’s pregnancy was outed to her parents through targeted advertisements (Hill, 2012). Recognizing these risks prior to data collection is critical for ensuring that analyses are ethical.

When ethical issues are apparent, the risks need to be weighed against the potential benefits. Researchers engaged in a particular data collection or integration project are often poorly situated to conduct these assessments. In academic and many corporate settings, outside evaluations are built into this process through the use of institutional review boards. Additionally, in many cases, it is important to gather informed consent from individuals about whom data are being collected (or integrated). These considerations

have a basis not only in ethics norms but also in international human rights laws (Constantin, 2018).

SECTION 3: EFFECTIVE MEASUREMENT WITHIN THE DATA STREAM

Before data can be meaningfully integrated across streams, measurements of key parameters within each of the individual streams need to be harmonized. In other words, the data need to be put into a format such that the conclusions reached from individual streams are comparable to one another, the measurements share meanings, and the populations the data describe are compatible. Typically, this process involves an attempt to conduct data cleaning processes, to appropriately link the population for whom data streams were available with the target population, and to address missingness in the data that were collected. In many cases, there may also be multiple components that will be used for any given measurement, requiring that analysts use one of a series of techniques to generate these measures. Collectively, it is important to understand and estimate the influence of any sources of error that may need to be accounted for in analyses across datasets. While the process of producing effective measures may use one or multiple datasets, the process serves to improve the measurement from the individual data streams rather than to generate conclusions across them, which we address in Section 4.

3.1 Data Cleaning

Social data are often messy. Cleaning may take different forms depending on the type and nature of the data that are being considered (see e.g., Ilyas and Chu, 2019). Furthermore, data cleaning should be approached with attention to the conclusions to be reached.

Official records can have all sorts of errors, ranging from outdated information to mistakes made when data were copied from one form to another (e.g., voting and health records). Survey interviewers transcribing verbatim responses are prone to shortening and omitting words, potentially distorting the meaning of open-ended responses. Social media data are rife with misspellings. And observers may sometimes guess at key pieces of information. These errors can distort conclusions if they are systematic. Hence, one central process for almost all data analyses involves cleaning the data to be used. Even when data cleaning is not strictly necessary, checking the plausibility of the observations obtained and spot-checking individual pieces of data to assess accuracy, the potential for bias, and the operational issues in the data collection activities is always worthwhile.

survey will ideally include a broad range of population subgroups in the sample that serves as a microcosm of the larger group of interest. An effective official data source will be one that produces relatively equitable coverage of all relevant population subgroups. When the design of a data collection effectively approximates the population of interest, less work will be required to improve the accuracy of conclusions from that data, and there is less chance that analytical strategy will result in erroneous conclusions. Indeed, statisticians often consider the difference between inferences that can be made by design and those that are accomplished through modeling. While both approaches can be appropriate, it is possible to understate the errors introduced by a design or in the process of analysis fitting a model.

For gaps that remain after data collection, a variety of weighing and adjustment methods can be used to link any given dataset with a target population. Often, data that come from a study vendor already include some form of weights designed to accomplish this very feat. Understanding the methods used to generate those weights is often important as different strategies for weighting and adjusting datasets can yield somewhat different conclusions. Common methods for weighting data include post-stratification (discussed in the “Drawing Inferences” paper), raking (iterative proportional fitting), and calibration. These approaches require population benchmarks that a newly collected dataset can be pegged to. There are a number of textbooks and other how-to guides that lead researchers through the process of producing weights (e.g., Valliant, Dever, and Kreuter, 2019). An important note about using weights to adjust to a population is that the process of doing so may increase the overall variance in results obtained and therefore reduce the effective sample size (Kalton and Kasprzyk, 1986). This is particularly true as the variability of a set of weights increases. As the differences between a sampled population and the target population grow bigger, the weights assigned to different cases become more variable. Attempts to correct for bias by weighting can therefore increase overall error if the variance introduced through weighting is larger than the bias being corrected for. When this happens, it is sometimes preferable to forgo an attempt to formally weight the data to match the population and instead estimate the value of a particular parameter across multiple population subgroups and then assess whether those estimates are generally consistent with one another.

A series of alternate approaches to adjustment focus on generating inferences that match the population in other ways. They have been discussed in the context of nonprobability samples to which design-based inference does not apply. These methods can be classified into quasi-randomization versus superpopulation approaches (Elliott and Valliant, 2017). Quasi-randomization approaches attempt to model pseudo-inclusion probabilities using the available data (e.g., propensity score weighting in Lee and Val-

liant, 2009), and the weights are applied to all variables in the given dataset. Another approach involves thinking of any given sample as part of an arbitrarily large superpopulation with various subdivisions. When thinking about the sample in this manner, it becomes possible to use methods such as multilevel regression and poststratification (MRP [Wang et al., 2015]) or merging data with large official sources of data to generate inferences. Superpopulation approaches attempt to model conclusions specific to a given outcome rather than the dataset. Because these sorts of approaches use models to assess and estimate the links between sample and population, model misspecification becomes an important consideration in assessing the reliability of the conclusions reached. Overly precise models can mistakenly include random noise as a signal and lead to undue confidence in results, while insufficiently precise models can induce null results. Estimating precision accurately depends on understanding the sources of error in the data as well as how they should be parameterized.

One important note about linking data with the population is that the tools to conduct these links reduce bias at the expense of introducing additional variance into estimates. Because this is the case, it is important to consider the sensitivity of conclusions to both the reduction in bias and the increased variance. In many circumstances, the total errors in models that adjust data can be larger than the total errors in models that do not do so. For this reason, many analysts report population point estimates using weights but do not report weighted correlations or regressions (see Pasek, 2015).

3.3 Addressing Item Missingness (Imputation)

For missing data on some variables within individuals, a series of techniques have been developed to generate conclusions despite this missingness. These approaches are often referred to as imputation and are designed to identify values for data that might be missing. Imputing for missing data is important but can also be complicated. Disregarding cases with partial data, known as listwise deletion, is likely to produce estimates that are biased toward the kinds of individuals whose data is complete, meaning that this approach is distinctly suboptimal (even though it is the default behavior in statistical software packages [King et al., 2001]). Furthermore, doing so implicitly assumes mean imputation, as the missing cases are replaced with the mean of the observed cases at the inference stage. Providing each missing case with the most likely observation, however, introduces a different problem as estimates may become overly precise (as they are based on more data than they should be and that data has too little variability). Instead, proper imputation requires simultaneously estimating missing values and accounting for the errors in those estimates.

or summed, or the proper weighted combination of measures can be estimated using a technique called principal component analysis (Dunteman, 1989). Errors from individual measures are still present using these techniques, but they are less important because they are discounted by other measures that do not share the same errors. Estimates of the similarity of the different items (typically coefficient alphas or correlation coefficients) help to reveal how well the measures used hang together.

More sophisticated types of composite measures focus instead on the shared variance across a series of items, generating what are called “latent traits” using techniques such as exploratory or confirmatory factor analysis (Harman, 1976) and item response theory (Embretson and Reise, 2013). These methods produce similar composites to averaging methods, but their focus on shared variance means that idiosyncratic errors are completely ignored. Factor analysis and item response theory approaches can also generate weights and yield important information about how well the eventual measure captures the underlying construct of interest.

3.5 Validation

Validating each data stream before integration adds to the confidence in the results drawn from the integrated data. Reliability and validity are two elements that can be considered when engaging in this process. Reliability is a concept applicable for multi-item measurement instruments (e.g., satisfaction with life scale by Diener et al., 1985) and indicates whether items intended to measure the same construct yield similar results. Cronbach’s alpha (Cronbach, 1951) is undoubtedly a popular tool, but its assumptions are easily violated (Sijtsma, 2009). When response style (discussed in Section 2.7.3) is a factor, Cronbach’s alpha-based reliability measures may be erroneously inflated or deflated. Alternative tools such as the greatest lower bound (Jackson and Agunwamba, 1977) as a reliability measure may be less susceptible to such issues.

Validity can be assessed to some extent. For example, it is well known that life satisfaction is related to health (Grant, Wardle, and Steptoe, 2009), so we should expect a significant and positive relationship between measures of these two variables. This sort of correlation would provide formal evidence of convergent validity. In contrast, analysts may be interested in establishing discriminant validity in some cases where it may be important to show that items are measuring things that are distinct (e.g., a prime minister’s approval should be distinguishable from party voting intentions). Face validity is another tool to exploit for results when analysts have qualitative information. Life satisfaction that is too high or too low may be a signal for unfit validity. Other critical types of validity for measures include content validity (evidence that a measure captures all key components of a

complex concept) and construct validity (evidence that a measure is indeed measuring the targeted concept) and should also be considered. Shadish, Cook, and Campbell (2002) outline some of these key types of validity and discuss ways to assess them.

When validation can be done within each data stream and the results are consistent across streams, one can be confident about discerning when errors from different streams are likely to compound or instead cancel each other out. Validation exercises may add to the defensibility of data comparability, especially when data streams are from different cultural groups.

3.6 Linking Data Streams Together

After within-dataset issues have been addressed, the process of actually combining data streams to produce a single dataset often involves methods to match cases to one another at either the individual or aggregate level. There are a number of approaches to these matches that depend inherently on the nature of the datasets being combined. Data can be linked based on individual identifiers, household identifiers, group identifiers, location-based metrics, networks, or sometimes content-based metrics. Each of these types of linking is coupled with a level of precision and some amount of error. Failures to recognize these sorts of issues can be pernicious, as was the case when mistaken identity linking following the 2020 election led to inaccurate perceptions of voter fraud (Sherman, 2020).

Because there are several ways to link datasets, the quality of links and the rate at which links can be made can vary considerably. Links made using individual-level identifiers depend on the shared accuracy of those identifiers and the tolerance for discrepancies. In linking survey data with state voter records, Berent, Krosnick, and Lupia (2016) found that linkage rates could vary between 45 and 77 percent depending on the stringency of the match required. Stringent matching techniques produce highly certain matches, but match fewer records overall whereas less stringent approaches may err by linking individuals who are not true matches (e.g. separate individuals who share the same name) but match more records overall. Both can be sources of bias in analyses, as records are more likely to be mismatched if they contain common names and less likely to produce a match at all for individuals who occupy less privileged groups in society, for which recordkeeping may be less accurate. As an example, studies by the U.S. Census Bureau that attempted to link administrative records, census data, survey data, and third-party data such as consumer data show a lower match rate for racial and ethnic minorities, particularly Hispanics (Ennis et al., 2015). Individuals who rent their homes are less likely to be found in matching consumer data (Pasek et al., 2014), presumably because they produce fewer other records that get picked up by data collectors (e.g.,

warranties, official records, etc.). When data are connected across sources, individuals whose data can easily be linked are more likely to register as complete cases in the final datasets. This should be treated as a potential observational bias.

Links between datasets that are made using geographical, behavioral, or other aggregate-level connections can prove valuable for including contextual information as well as other indicators that may not be observable at the individual level. Analysts using these kinds of data must take care, however, not to overgeneralize. Individuals whose attributes differ from the norm in their group or area can sometimes be characterized incorrectly with aggregate-level data. This can be a particularly pernicious problem when the real influence comes from relative, rather than absolute, positioning. For example, a middle-class individual living in a poor neighborhood may act in ways that differ from what might be expected from their income level as well as from their geographic location. Analyses that fail to consider these sorts of relative effects can use linked data to reach precisely the wrong conclusions or may reach correct conclusions for the wrong reasons.

Section 4. Analyzing Data Across Streams

There are many ways to conduct analyses of integrations of data, and no single approach is uniformly ideal. Once the issues within data sources have been addressed, analysts can work toward analyses that bridge across data streams. The approaches used depend on the goals of a particular project. If the principal goal is to come to an understanding of a phenomenon that is measured in multiple data streams, then ensemble approaches are typically used to generate estimates from the integrated data. A common example of an ensemble approach is the use of surveys, betting markets, and expert assessments to predict the winner of an election. These sorts of approaches can employ a variety of strategies to pool estimates and increase precision, both in aggregate and for subsets of the data. If the key question is answered by comparing different datasets, models that address multiple contexts (such as populations, cultures, times, or environments) may be ideal. Various approaches allow researchers to recognize the extent to which errors in some circumstances may be idiosyncratic versus part of a larger notable pattern. By accounting for variability due to consistent sources of error in subsets of the data (e.g., firms in survey data, temporal dynamics in time series data, proximity in geospatial data), analyses can distinguish signal from noise. Additionally, when the goal of a particular analysis is the identification of a causal relationship, processes that allow researchers to identify time-ordered effects and to rule out known sources of spuriousness can yield stronger conclusions. A series of methods for each of these goals is presented in Table 3D-3 (Parts 1, 2, and 3 respectively).

protest. Evidence of causality would be stronger if willingness to protest increased shortly after the policy shift. Such a link would also be evidenced if willingness to protest changed across multiple locations after the policy rollout in each of those areas.

4.3.3 Causal Evidence Through Theory Confirmation

Finally, many claims about causal relations include not only an assertion that some cause leads to some effect, but also a theoretical set of mechanisms whereby that link occurs. For instance, there may be an expectation that protests will result in greater support for an insurgency when protests are reported on by media outlets and when protesters are met with repression. Evidence of these sorts of conditional effects (often referred to as mediators and moderators) can bolster a causal claim because observations suggest not only that the key variables are correlated but also that the relation between those variables corresponds with other expectations that are unlikely, absent such a causal process. There are ways to formally test for mediation and moderation that can help confirm causal assertions, though these formal tests (particularly for mediation) often depend on assumptions that are difficult to test (Hayes, 2009).

REFERENCES