credible, and reliable manner. In trying to respond quickly, however, an intelligence analyst may overlook some key methodological aspects that determine the quality of survey results, which varies depending on chosen methods. It is thus relevant for intelligence analysts to take the time to ask the right questions to assess the key survey quality elements.

Intelligence analysts may consider their research approach as an activity to fill gaps in their knowledge, the purpose of which is to conduct a well-rounded study of a public opinion survey. It is thus recommended that intelligence analysts gather and study both contextual information (the survey background) and technical information (the survey design and results) to establish both the credibility and soundness of the survey. Definitions of these concepts are provided as follows and summarized in Figure 3A-1. Also, Figure 3A-1 introduces key questions that can help guide analysts in their research inquiry.

Contextual information is the “metadata” or background information that helps situate the survey in place. It essentially includes descriptive information about the study (not the content of the survey design or the survey data themselves) that can help intelligence analysts learn about the study. Contextual information alone is not enough to accept survey results, but it helps the analyst discover the foundation of the survey. A survey’s context can be categorized into seven analyzable elements:

1. Survey purpose
2. Survey authenticity

3. Professional integrity
4. Survey sponsorship
5. Survey research team
6. Background of survey firm
7. Reporting and dissemination

Technical information can be thought of as the properties of the study that delineate the planning, scope, adequacy, and limitations of the survey to represent a population of interest. Technical information is typically what is discussed by survey methodologists and sampling statisticians when creating, assessing, and interpreting high-quality surveys. Technical information can be categorized into four elements:

• Sampling properties
• Coverage properties
• Nonresponse properties
• Measurement properties

Survey credibility refers to the perceived quality of contextual information that makes the survey worthy of belief (i.e., the “believability” of the study). The analysis of contextual information can help to better assess whether the survey results are likely to be “believable,” broadly defined. The importance of contextual information cannot be overstated as this information can tell us a great deal about the credibility of a study.

Survey soundness refers to the quality assessment of technical and scientific elements from a survey methodology perspective. This information helps to determine whether a survey is technically viable for intelligence analysis (the scientific “defensibility” of the study). The assessment of technical features can help the analyst to assess whether the results are likely to be “representative.” It is important to note that there is a distinction between whether the survey is of sufficient quality (“defensible”) to make a statistical inference as opposed to the inference itself. The soundness of a statistical inference is also an important consideration and depends on the combined quality of all technical information. Typically, this technical information is analyzed through a survey methodology paradigm known as “total survey error” (discussed below in “Technical Information: Total Survey Error Assessment”).

A five-star quality rating tool is recommended to help intelligence analysts to quickly assess contextual and technical information of a survey. A rating system can help analysts determine the extent to which contextual information indicates the study is credible and the extent to which technical information indicates the survey data sound. Each of the elements introduced in Figure 3A-1 can be rated and enable more holistic assessment.

tions a survey result does not mean that the survey is true or valid. If the intelligence analyst cannot easily identify the original source of the survey, it is better to treat the survey result as not credible.

In other situations, the source can be traced back but the group or individual who created the survey is not legitimate or credible. This is particularly problematic during election campaigns. The intelligence analyst needs to be aware of inconsistencies in the field. Sometimes a verified, authentic survey may provide results that are deemed inaccurate and get quickly dismissed, while a fake survey could be perceived as accurate and end up being frequently cited. Illegitimate results can enter the political debate and be cited by high-profile individuals in persuasive arguments designed to swing public opinion in a particular direction.

While it is tempting to use surveys of dubious credibility (especially if many others are citing the results), it is preferable that intelligence analysts differentiate between verified and unverified surveys and assign lower value or no value to unverified surveys (Figure 3A-3).

___________________

¹ https://www.aapor.org/Standards-Ethics/Survey-Practices-that-AAPOR-Condemns/SourcesCited-in-Best-Practices-Survey-Practices-t.aspx.

² https://www.aapor.org/Standards-Ethics/Survey-Practices-that-AAPOR-Condemns.aspx.

“Horizon 2020” and “Horizon Europe”), government agencies that fund national strategic studies (such as the Academy of Finland), or ministries in countries proving direct public funding for contractual work guided by concrete objectives (like Taiwan’s Ministry of Science and Technology). Other groups such as nonprofit organizations may partially or completely fund research that aims to inform strategies for improving the future of local or national communities. Public entities can provide financing for public opinion research on behalf of political parties (Mexico’s National Electoral Institute, for instance, provides financing to political parties to conduct their own research).

There are other types of informal funding organizations that may operate with greater discretion in how they award the work and distribute data. A business, for example, may want to quickly assess the political and electoral landscape in a country during an election campaign before committing support to a particular candidate, so it may commission its own survey. When results of such surveys become known to intelligence analysts, they represent a challenge to interpretation since sometimes the survey is designed with a limited scope in mind (e.g., trying to answer a particular question based on narrow, private interests). Sometimes the sponsor may be seeking the data to inform personal decisions, but at other times they may be seeking to disseminate the results anonymously in order to promote an agenda. Those with an agenda in mind may use different tactics to achieve their purpose. They may, for example, partner with a news organization that may not have enough financial resources to conduct its own surveys and is willing to engage in a friendly agreement to work with the sponsor and publish the survey results if they are granted the exclusive rights for publication of the results. Surveys published using these tactics are difficult to identify, but they do exist. When the identity of the sponsors is unclear, the credibility of the study becomes questionable.

Beyond financial disclosure, there are other methodological concerns related to perceived sponsorship. The survey methodology literature shows that the stated or perceived survey sponsor may make a difference in survey quality. A survey conducted, for example, during the highly polarized 1990 Nicaragua general election in which Violeta Chamorro unexpectedly defeated incumbent Daniel Ortega demonstrated that respondent perceptions of partisanship on the part of survey researchers may have influenced the respondents’ answers. Many such surveys failed to correctly predict the election outcome. An experiment led by academic researchers from the University of Michigan during the Nicaraguan campaign shed some light on the issue. Academic researchers experimentally manipulated the color and design of pens used during the interviewing process in a face-to-face survey. One experimental condition used colored red and

black pens with the legend “Daniel Presidente.” A second condition used white and blue pens with Chamorro’s party acronym. A third condition used a red and white pen with no markings (suggesting neutrality). The survey protocol of the experiment did not call the respondents’ attention to the color of the pen being used by interviewers. Instead, interpretation was left up to the participant. The experiment yielded results suggesting that respondents may have derived meaning of the perceived sponsor of the study based on the visual cues (Bischoping and Schuman, 1992). Perceived or stated sponsorship of a survey can impact results, and the analyst needs to search for even minimal clues that can help establish limitations of a study (Figure 3A-5).

while several groups (for instance vendors) may have different standards for quality.

Sometimes when there is high demand for data collection services (for instance, during election season or when major events are happening such as decennial censuses), the number of available companies capable of offering manpower services decreases, and so the quality of contractors varies. In other words, the few companies that can carry data collection operations may already be busy, opening the door to the use of less experienced vendors (such as local groups or university students) available to more established social researchers to conduct surveys.

Lack of information about who the research team is makes it harder to establish credibility for the study, thus making the rating score lower. On the opposite end, when the identity of the survey research team is known, the score on credibility can be higher (Figure 3A-6).

___________________

³ https://www.cnn.com/2019/07/09/politics/read-cnn-transparency-questionnaire-polling/index.html.

into probability and nonprobability methods for sampling. The probability approach supports statistical inferences about the population based on parametric statistical theory and allows calculation of a theoretical margin of error. Nonprobability approaches allow more flexibility in some situations but do not directly support statistical inferences about a population (some theoretical assumptions are not testable). In this section on sampling error, we first review the probability approach and then we briefly discuss the nonprobability approach.

Probability Sampling

Survey methods that rely on a random selection of individuals are designed to support inferential statistics. Probability studies that can be generalized to the entire population of interest report confidence levels and boundaries for error. Relative to nonprobability approaches, the main differentiation is that probability methods are based on statistical theory that allows more clear estimates of the sampling error associated to them. Inferential statistics is based on the idea of multiple, repeated possible samples from the same population. The theoretical distribution of all possible samples (out of which a statistic of interest such as a mean is estimated) is the basis for estimating the possible sampling error. This potential variability in possible samples is known as the standard deviation of the sampling distribution (Figure 3A-10), which in turn is the basis for calculating the sample variance (and standard errors). Such variance allows the computation of the theoretical margin of error in a survey. The statistical theory behind the calculation of a margin of error gives us statistical confidence that results support inferences about the parameter of interest, even when the

sample that we happen to be observing does not sit exactly in the middle of the sampling distribution (Kalton, 2020; Lohr, 2021). In other words, even well-designed surveys can yield estimates that may not be right at the center of the theoretical sampling distribution and yield seemingly “unsound” results why is which a margin of error is calculated.

Inferences based on probability samples are possible because members of a population have a known, non-zero probability of being selected, and they are chosen at random. In probability studies, the type of sampling strategy and (lack of) proper weights may define the bias and variance in a result. A bias might occur, for example, when public opinion studies disproportionally select subgroups in the population to represent certain groups or regions. Let’s say a hypothetical survey obtains a representative sample of the country of Mexico, and the survey has 500 interviews from Mexico City, and 500 cases from the rest of the country (a total of 1,000 cases). In this hypothetical study, the researcher systematically oversampled individuals from Mexico City. To be able to make appropriate inferences for the whole country (that is, to have unbiased results), the base weight needs to account for the sample strategy, that is, whether the case came from Mexico City or the rest of the country. In other words, the 500 cases from Mexico City can be representative of Mexico City themselves, but to make an inference about the entire country based on the 1,000 collected interviews, the 500 cases coming from Mexico City need to be weighted down so they reflect the appropriate probability of selection. If they are not weighted down, the Mexico City cases will contribute too great a weight to the overall estimate, leading to results that are biased toward Mexico City.

In probability surveys, the number of cases and other factors such as the variance of the weights determines the margin of error. The general notion is that the higher the sample size, the lower the margin of error will be. At the same time: the intelligence analyst needs to keep in mind three important aspects: (a) the relationship between sample size and margin of error is not linear, (b) the size of the population typically does not have a major influence on the decision about sample size, and (c) a study with a large sample size also means a more challenging field operation. Figure 3A-11 shows the relationship of sample size (x axis) versus the margin of error (y axis). Typically, an increase from, let’s say, 400 cases to 1,000 cases means a reduction in margin of error, since the margin of error associated with 400 cases is +/–4.9 percent, and for 1,000 cases it’s +/–3.1 percent under simple random sampling and a 95 percent confidence interval. The relationship shown in Figure 3A-11 holds irrespective of the size of the population. There are other more advanced sampling techniques that incorporate stratification and clustering, which affect the properties of the margin of error. Survey designs that involve dividing the entire population into mutually exclusive segments (strata) and collecting data based on natu-

rally occurring groups (clusters) are based on multistage stratified cluster sampling techniques. The more advanced sampling techniques are typically referred to as complex random sampling and aim to have better (lower) bias and variance properties (see Lohr, 2021).

Regardless of the sampling technique, whether simple or complex, the gains in a margin of error resulting from a bigger sample quickly decrease to the point that there is just a marginal difference. For example, the difference between a study of 2,500 cases (+/–2.0 percent) and a study of 5,500 cases (+/–1.3 percent) is less than 1 percentage point under simple random sampling. More importantly, a data collection operation with many more cases requires more careful supervision of the field work and more support for the field staff. For studies that are not carefully planned or executed, a large sample size may mean having higher chances for greater error (including fabrication or falsification of interviews and inadequate field supervision). Sometimes, it is preferable to have a carefully conducted and supervised study of, let’s say, 800 cases than a rushed and poorly supervised study of 8,000 cases. This is not to say that studies with large sample sizes are always low quality. A study may aim to measure subregions in a country or subgroups in the population (for example, 800 interviews per state in a country with 10 states may result in a study of 8,000 cases), and this can be carefully conducted to maintain high quality in the study.

At the other extreme, the analyst needs to be aware of questionable surveys that claim (or may collect) a high number of cases just for the sake of claiming a much lower margin of error in hopes of gaining greater public credibility. Such a claim does not always translate into a high-quality study. It is recommended that the intelligence analyst try to identify what may explain the motivation for a large sample size to assess the quality of a survey.

___________________

⁴ For more details on nonprobability methods, please see the companion paper “Alternatives to Probability-Based Surveys Representative of the General Population” by Ashley Amaya, provided later in this Analytic Framework.

may ask interviewers to select respondents based on age and gender instead of using a random selection mechanism not controlled by the interviewer. Interviewers are tasked with filling pre-established quotas until the desired number of interviews are completed. At the end of the study, the final demographic composition is likely to look very similar to the distribution of cases in the population, but the problem is that the selection of cases was not based on a random protocol; it was left up to the interviewer to decide. Interviewers may have selected cases that were more willing to voice opinions in a survey. This subjective selection challenges the assumption of random sampling and limits the ability to draw statistical inferences with confidence. When a survey uses a within-household respondent selection procedure that is not based on probability principles (such as the quota sampling approach described here), the quality of estimates is likely to be downgraded, and the analyst may assign a lower rating to the study in question.

There are nonetheless situations in which quota sampling may represent an opportunity to study public opinion in under-researched populations. It is well known, for instance, that there are many restrictions in China on conducting public opinion research, but China offers more flexibility to conduct market research. A group of researchers at Hong Kong Polytechnic University conducted two quota-sampling surveys in mainland China between 2018 and 2021. Their objective was to understand engagement and behaviors on social media platforms like Weibo or Wechat with political connotations (e.g., critical and satirical social media comments) among Chinese citizens with online presence (“Chinese netizens”). The researchers implemented an online survey utilizing an online market research panel through quotas based on gender, age, educational level, income, and province. They replicated in the survey questionnaire possible behaviors on social media (recommending, reposting, or skipping certain opinion comments). While their study may not have allowed full inference about the Chinese population, it provided important insights on political behavior in social media environments (Wong and Liang, 2021).

There are circumstances that merit a mix of probability and nonprobability methods. These situations are usually encountered when researchers try to study hard-to-reach populations, such as immigrants, homeless individuals, those resisting an established authority, ethnic minorities, individuals in extreme poverty, survivors of domestic abuse, individuals diagnosed with a new disease, individuals engaged in illegal activities, and individuals with a particular language. For these groups, strict probability surveys may not be entirely feasible or possible. It is thus advisable for the intelligence analyst to understand the motivation to produce use combined methods, as not all studies of hard-to-reach populations are the same (Kalton, 2009).

(a minority in a conflicted society) conducted in Israel in the early 2000s utilized the 1995 census information made available to survey researchers by the Israeli Social Science Data Center at the Hebrew University of Jerusalem, which was supplemented by the official list of eligible voters and Central Bureau of Statistics records. This information (organized by region, municipal status, and composition of population group) was the basis for selecting localities as primary sampling units under a multistage stratified sample (Gordoni and Schmidt, 2010).⁵

When some members of a target population are systematically excluded from the sample frame, there is a risk of coverage bias. Coverage bias can occur in two forms, as over-coverage (when there are units in the sampling frame that do not belong to the target population) and under-coverage (when the frame is incomplete and misses important parts of the target population). Undercoverage is far more problematic than over-coverage (people without phones, for example, being excluded from a phone survey), since over-coverage can be addressed easily by excising redundant sampling units. Under-coverage, which is more challenging, is frequently addressed through the combination of multiple sample frames. Combining several frames, nonetheless, requires estimation of a joint probability of a sample unit belonging to several frames. Obviously, the combination of frames assumes the existence of multiple sample frames. Often, in practice, new sample frames need to be developed, which can be costly and time-consuming (Brick, Cervantes, Lee, and Norman, 2011). A study based on multiple frames requires the creation of special selection weights, as there is an increased uncertainty in results associated to the possibility of selecting the same unit multiple times; as such, the level of theoretical variance increases for the study (Figure 3A-13).

There are trade-offs around the selection of sample frames. A hypothetical survey researcher, for example, may want to conduct a telephone survey in a country in which neither landlines nor cellphone telephone numbers perfectly cover the entire territory. The researcher may aim to produce unbiased estimates using a dual-frame sample approach (landlines and cellphone numbers) at the expense of higher variance (variance being due to the multiple-selection weights). Another hypothetical researcher in the same country may decide to reduce the variance by not using multiple frames and focus only on a single frame for the study (excluding parts of the population). There are researchers who prefer to disclose the limits of a study (that is, recognize that they use a single frame) and avoid the need to develop multiple sample frames.

Decisions on sample frames are frequently based on practical field conditions and costs. In that sense, just because a sample frame is based on

___________________

⁵ For a more recent example, see https://www.pewresearch.org/fact-tank/2016/03/08/keyfindings-religion-politics-israel/.

geographic information like household addresses does not mean the study is automatically better than, say, a telephone survey or even an internet survey. In high crime areas, for instance, respondents may not be willing to answer the door to a stranger (i.e., an interviewer), or interviewers may not want to visit the area due to safety concerns. Perhaps both interviewers and respondents have concerns such as public health conditions, as in the early months of the COVID-19 pandemic in 2020. In such situations, survey interview respondents may be more easily reached with a phone call without having to spend a lot of money in a communication campaign to create an awareness of the study or to develop protocols for a safe interviewer-respondent interaction. Similarly, just because a phone call is a more convenient form of communication does not necessarily mean that a telephone survey offers better coverage than an area probability survey. Social context and practical conditions in a country frequently influence

researchers’ decisions about sample frames, and the analyst needs to investigate and assess the reason. Analysts need to identify the trade-offs made by survey researchers on a sample frame used to represent a population of interest. Understanding why a researcher chose a certain sample frame provides insight that can help determine what kind of statistical inference can be made.

Nonresponse Error

Probability sampling theory relies on having a representative sample selected through a random mechanism. When the sample size is reduced at random, the uncertainty, or the variance translated into standard errors and then converted into margin of error, increases. When the sample size is reduced due to a systematic reason, there is a risk for bias. To understand nonresponse and its effects, it is helpful to know how it is studied by survey researchers. The survey literature has classified nonresponse in three major categories (which are accepted by several professional organizations): (a) noncontacts (when the respondent can’t be located), (b) refusals (when contact was made with the selected individual, but the respondent declined to participate), and (c) other reasons (such as language barriers, health issues, or death) (Groves and Couper, 1998; Singer, 2006). In that sense, nonresponse due to lack of contact is usually about an individual’s inaccessibility or unavailability, that is, contactability. Nonresponse due to refusal is about an individual’s willingness to participate. Nonresponse due to other miscellaneous reasons is usually due to some form of inability.

In some instances, the systematic reasons leading to nonresponse are clear and allow the identification of potential bias. If a survey aims to study literacy levels in a population, but the mode of survey administration is a self-administered mail survey, for instance, individuals with low or no writing abilities would not be able to participate. In this example, individuals are unable to participate even if they are successfully contacted and willing to do so, and this would lead to bias. In opinion surveys, the reasons for nonresponse are less clear and often mixed, but most of the time the reasons are related to respondent willingness. Regardless of whether nonresponse is due to contactability, willingness, or inability, the idea remains the same: When there is a systematic relationship between the statistic of interest and nonresponse, there is a chance of a biased result (Figure 3A-14). Establishing the systematic effects of nonresponse is hard, and analysts could draw from previous literature or theory to understand the impact of nonresponse on survey estimates (as it is discussed in the following paragraphs).

Several professional organizations provide standardized ways of calculating response rates, which usually make use of the three elements described in the previous paragraphs. Many international organizations

follow definitions proposed by the Council of American Survey Research Organizations and the American Association for Public Opinion Research.⁶ While a response rate is helpful in understanding levels of survey participation in a study, it should not be taken as direct measure of survey quality. It is often more helpful to gain information about the potential reasons that may have caused the response rate in a study than the rate itself.

Admittedly, it is not always possible to clearly establish the relationship between nonresponse and the statistic of interest (Groves and Peytcheva, 2008; Groves et al., 2006). Nonetheless, examining individual elements may shed some light on the reasons for nonresponse. Examining aspects related to contactability, for instance, may help to partly understand non-

___________________

⁶ https://www.aapor.org/Education-Resources/For-Researchers/Poll-Survey-FAQ/ResponseRates-An-Overview.aspx.

response. Contactability aspects refer to the issues related to the inability to locate the selected individual for a survey, the individual being too busy at the time of contact to participate in the survey, or a lack of technological means for reaching the individual (such as telephone service, internet access, email account, city-style residential address). Aspects related to willingness to participate are somewhat different, and they include perception of the study topic (e.g., as unexciting or sensitive, concerns about privacy; perceived lack of study legitimacy; general antipathy to surveys; hostility toward the government; perceptions or views about the sponsor; crime rates; fear of strangers; cultural, religious or political circumstances; and intergroup differences. The unwillingness to participate can take two forms: unit nonresponse and item nonresponse.

Unit Nonresponse

This occurs when some selected individuals choose not to participate in the entire survey. To minimize unit nonresponse, survey researchers usually develop strategies that rely on recruitment materials explaining the purpose of the survey, delivering small gifts, offering several modalities to participate (phone, web, postal survey, or in-person), and offering monetary or in-kind incentives. These strategies may have different levels of success depending on contexts.

Despite best efforts, nonresponse is likely to occur in any survey, as there is evidence for a decline in survey participation around the world (De Heer and De Leeuw, 2002; Luiten, Hox, and de Leeuw, 2020). The issue of unit nonresponse is typically addressed through statistical weighting.

Intelligence analysts need to be cognizant of the fact that survey weighting may help reduce the potential bias when or if the demographic variables used for weighting are related to survey participation and the statistic of interest (Kalton and Flores-Cervantes, 2003; Kalton, 2020), but weighting can introduce more uncertainty to the estimates in the form of added variance. Indeed, the effect of weighting on the statistic of interest is calculated through what is known as weighting design effect, which is essentially the impact of differential probabilities on the estimate throughout the survey selection process (Henry and Valliant, 2015; Spencer 2000). The weighting design effect is further affected by another design effect called sampling design effect, which comes from the use of strata and clusters in complex random sampling (West and McCabe, 2012). When the design effect (the “deff”) is 1, the variance is similar to one obtained under simple random sampling. When the overall design effect is greater than 1, though, the precision of the estimate is likely to decrease (in other words, the variance is likely to increase). It is rare when public opinion surveys report the deff for a variable of interest, but when they do, the deff can be taken as a positive

signal of a study that aspires to be transparent about its limits. The analyst may want to use the deff as an additional piece of insight to understand uncertainties about a public opinion survey.

In recent and advanced methodological discussion around nonresponse (and how to address it), response rates are becoming less relevant, and the notion of representativeness is starting to become more relevant. Representativeness measures how successful the survey has been at representing subgroups of the population based on probability mechanisms and not predefined quotas. The criticism to assess surveys based on response rates is that it is relatively simple to increase response rates, since a researcher may collect data from the “easy cases” (individuals willing to participate), and then rely on substantial weighting to bring demographic distributions to conform population controls. This creates, however, a clear risk for nonresponse bias. In other words, a response rate should not be taken as an indicator of survey quality (for example, a response rate of 60 percent does not necessarily mean that it has superior quality over a survey with a response rate of 40 percent). Also, recent research suggests that sometimes a higher propensity to participate may lead to measurement error (Bach, Eckman, and Daikeler, 2020).

Recent survey studies have started to utilize methodological tools that rely on real-time monitoring of sample composition. These tools are known as “response indicators” or r-indicators and are based on predictive modeling of survey participation (based on key predictors such as region, age, gender, marital status, employment status, health insurance, and homeownership status) while the survey is in the field (Bethlehem, 2012; Schouten, Cobben, and Bethlethem, 2009; Wagner, 2012). The idea is to selectively intervene depending on whether the survey is falling short on certain demographic subgroups. The types of interventions during the period of data collection can range from more aggressive monetary incentives to tailored recruitment materials for certain subgroups, the subgroups likely to be underrepresented in the sample (Schouten and Shlomo, 2017). These are ideas that show promise as tools to address survey nonresponse. The notion of r-indicators is provided here to give the analyst something with which to navigate the gathering of intelligence around nonresponse, but the information and resources needed to develop predictive modeling (and real-time intervention) is not the same across countries at this point in history, and it is not a common feature in international public opinion studies.

Item Nonresponse

This occurs when respondents do not provide a response to a specific question. This is another source of error that can occur either systematically (bias) or at random (variance). Some survey items may experience higher

nonresponse expressed as “don’t know,” “can’t choose,” “prefer not to answer,” “I don’t remember,” or simply skipping the question on web surveys. When a question has a high item nonresponse rate, it may be signaling that the survey question may be too sensitive or too complex to answer, lack the expected response choices, show unfamiliarity with the topic, use poor wording, or seem to want to elicit undesirable answers. Some of these item nonresponse issues are anticipated by survey researchers and resolved through the improvement of the questionnaire using psychological and cognitive theory (discussed below).

Survey researchers commonly address the issue of item nonresponse through statistical methods. One simple approach is to only use respondents who provided a response in the analysis. This approach is usually known as list-wise deletion. Using only the answers from participants who systematically provided a response, though, risks over- or under-estimation of a statistic of interest (bias). There are other better statistical methods used to address item nonresponse, and they can be classified as class weighting, single imputation, and multiple imputation methods.

Class weighting (a form of basic modeling) attempts to reduce bias. Class weights are weights applied to cases on top of the weights outlined in Figure 3A-15, which further increases the variance. Single and multiple imputation approaches rely on assigning plausible imputed values to missing values that were not collected. Multiple imputation methods predict the probability of responses being missing and estimate plausible values for the unobserved data based on the observed survey data and auxiliary data that might be available from the sample frame. Unlike class weighting and single imputation methods, multiple imputation methods already consider the uncertainty (variance) associated with the fact that actual data from nonrespondents were not collected.

In the imputation theoretical framework, missing data are assumed to come from a distribution that is modeled using available information to recreate the correlation structures of the analyzed data (Little and Rubin 2019; Rubin, 2004). Compared with the list-wise deletion approach, class weighting and single imputation methods are theoretically superior since they take some practical steps to address data missingness, but multiple imputation is superior to single imputation. The intelligence analyst may nonetheless see the use of list-wise deletion approaches in reports more frequently than the use of single or multiple imputation methods.

Measurement Error

The notion of measurement error is based on the difference between a true value and an estimated value. Measurement error occurs when the survey measure (i.e., the survey question) does not measure the intended construct of interest. This can systematically produce bias if, for instance, there is a leading question steering respondents in one direction, or randomly increase the variance if, for instance, a survey question is so ambiguous that it is unclear how participants interpreted the question. In general, it is easier to assess measurement error for factual questions than it is for subjective questions. Questions about the number of visits to the family doctor in a year, family income, the number of children, age, education, ownership of a smartphone are constructs that can have an external source for validation. This measurement error assessment is more difficult to quantify for nonfactual information, although there are methods that have produced scholarly research geared toward accomplishing this (Alwin, 2007).

Measurement error is more challenging to assess when interpreting results of subjective questions, such as questions on opinions, attitudes, motivations, beliefs, culture, religion, fears, anxieties, political preferences, ideology, and group identity. While it is not a straightforward task, features of the survey design can help the analysts review the potential for measurement error. The typical sources of measurement error stem from (a) aspects of questionnaire design (such as type of questions, question phrasing, response options, survey instructions, order of questions), (b) the questionnaire format (self- or interviewer-administered), and (c) the method of survey administration (face-to-face, mail surveys, computer-assisted telephone interviewing or CATI, Short Message Service or SMS, internet, automated interactive voice response, outbound automatic robocalls, social media platforms like Twitter and Facebook, and redirected inbound call sampling; see Figure 3A-16).

the question is presented as open-ended or closed-ended), and the order in which questions were presented (known as context effects). Variations to these elements can produce different results even when the rest of the survey design elements remain the same.

Researchers from the European Social Survey (ESS), for example, designed an experiment to document how changes in question wording produce different results in expressed attitudes toward gay men and lesbians. Fielding an experiment in Portugal, Hungary, and United Kingdom in 2015, ESS researchers administered three versions of the same question to different respondents. The introduction script was the same for the three versions of the question: “To what extent do you agree or disagree with each of the following statements?,” but the test statements were slightly different. Version 1 said, “Gay men and lesbians should be free to live their own life as they wish”; version 2 said, “Gay men should be free to live their own life as they wish”; and version 3 said, “Lesbians should be free to live their own life as they wish.” Analysis of the experiment showed that respondents expressed more positive attitudes toward this construct in version 3 (“lesbians”) than in version 2 (“gay men”) or version 1 (“gay men and lesbians”) (Kuyper, Sommer, and Butt, 2018).

Some questions are just too sensitive in certain contexts to be asked directly to respondents. In those cases, there is a chance that respondents will not provide candid answers. In 2008, for example, a group of political scientists investigated whether voters in Nicaragua had received gifts or favors in exchange for their votes during municipal elections. When the question was asked directly to respondents, about 3 percent of respondents admitted that they had received vote-buying offers, but when the question was measured indirectly using a technique known as “list experiment,” about 24 percent of respondents reported vote buying (Gonzalez-Ocanto et al., 2012). In general, sensitive questions can take multiple forms, and there are multiple methods that survey researchers can use to measure them including indirect questioning, randomized response techniques, item count techniques, nominative technique, forgiving wording, and endorsement experiments (Chaudhuri and Christofides, 2013). It is advisable that intelligence analysts take some time to find out whether questions presented in the questionnaire were sensitive and what methods were used to ask them. The more basic questions for analysts to guide their inquiries are presented in Figure 3A-17.

Cognitive Aspects of Survey Methodology

Another helpful theoretical framework to further study the cognitive, psychological, and communication part of measurement error is known as the cognitive aspects of survey methodology, or CASM. Survey methodolo-

gists rely on a 4-stage model to assess the psychology of survey response. This model proposes that respondents first try to make sense of the question wording (comprehension), then recall information from long- and short-term memory (retrieval), assess what information is relevant to deliver a response (judgment), and find response categories that express their response (mapping). Some problems related to questions are due to lack of question comprehension (faulty wording), but other problems crop up because questions are not easy to answer due to the reference period (memory erosion).

In other instances, the question was worded clearly and the memory of the event (or topic) is clear but the respondent is unsure whether to candidly report the information because of contextual elements. In still other instances, the respondent is willing to provide the information, but the response categories may not reflect well what the respondent wants to express. All these are forms of measurement error. The analyst is encouraged to review and use the four-stage model to understand the psychology of survey response to take a deep dive from the cognition and communication perspective (Tourangeau, Rips, and Rasinski, 2000; Willis, 2004, 2018).

Interviewing Format

Responses to survey questions can be affected by the presence (or absence) of an interviewer. If interviewers are present, they can help to motivate respondents to take the survey, explain the purpose of questions, clarify terminology, and thank them for their careful thoughts. At the same time, the presence of interviewers may present a barrier to truthful answers on sensitive questions, since respondents may experience some form of normative pressure to answer in a particular way. Female respondents, for example, may not feel comfortable disclosing sexual or drinking behaviors to male interviewers, and those survey items may either elicit a disproportionally high level of item nonresponse among women, capture insincere responses among women, or both. In that sense, interviewers may be a source of random (variance) or systematic error (bias), depending on the topic.

One typical household survey was conducted by the World Bank Group in the Southeast Asian nation of Timor-Leste to measure consumption-based welfare. The survey also included questions on access to justice (with similar questions to the Afrobarometer and Latinobarometer), as well as items on corruption, law and order, and women’s rights. Using advanced statistical techniques (multilevel modeling), a researcher affiliated with the World Bank studied respondent-interviewer interactions and found patterns suggesting that female respondents were more likely to be influenced by the perceived opinion of the interviewers than male respondents (Himelein, 2016). There is indeed a growing literature on interviewer effects suggesting that interviewer presence may have significant impacts on results. Such effects depend on respondent and interviewer characteristics such as race, sex, gender, perceived opinions, or affiliation (Mangione, Fowler, and Louis, 1992; O’Muircheartaigh and Campanelli, 1998). Obviously, determining interviewer effects takes considerable effort and resources, and intelligence analysts may not have all the necessary tools at their disposal (specialty statistical software or the luxury of time, for instance), but it would be worthwhile to ask those who conducted the survey to see if there

are reasons to believe that interviewer effects are a source of systematic measurement error in a particular study (Olson et al. 2020).

Method of Survey Administration

Intelligence analysts may want to check with regional survey experts which is the more prevalent mode of data collection in a specific country. But in general, at this point in our history, household surveys conducted with a face-to-face interviewing methodology continues to be a preferred method around the world relative to other forms of survey administration because it is thought to assure adequate coverage of the population. While face-to-face surveys may allow researchers to verify addresses and motivate respondents to participate, survey researchers sometimes use other methods that are much faster and less costly and that minimize interviewer presence, therefore reducing measurement error due to the interviewer.

There are advantages and disadvantages to using methods other than face-to-face interviewing. Take the case of telephone surveys, which are based on probability sampling (known as random digit-dialing) and conducted based on computer-assisted instruments known as “CATI systems” (computer-assisted telephone interviewing). One advantage to CATI approach is that it allows telephone surveys to avoid direct interviewer presence in a respondent’s home and thus may provide a less intimidating way to collect information. Telephone surveys may still provide some interviewer presence to clarify terms in the survey, and facilitate the possibility to program more easily experiments with different question wording (and thus better assess measurement error due to phrasing). The obvious tradeoff is the limited population coverage, as there may be members of the target population that have neither a landline nor a mobile phone. Even if measurement error was reduced with a telephone survey, though, coverage error would increase.

There are other methods that may bring advantages to survey researchers depending on research objectives, but there are also trade-offs around them. Internet-based surveys, for example, may allow the use of graphics and animations that would be hard to produce in other modes such as telephone or paper-based instruments. Researchers from NORC at the University of Chicago and Argonne National Laboratories (operated for the United States Department of Energy by UChicago Argonne, LLC) conducted a survey to study public behaviors on evacuation traffic patterns in the event of a major no-notice human-caused event in Chicago. The researchers designed an internet-based survey utilizing Google Maps in which respondents were instructed to drop pins to place themselves and family members on the map on a selected date. The survey instrument visually displayed hypothetical emergency scenarios on Google Maps and made

them available to respondents. The emergency scenarios randomly allocated time, location, and event severity across the Chicago metropolitan area, as well as event radius and government recommendations. Respondents were asked to specify their travel behavior under the emergency circumstance by dropping a pin on their destination location. The dynamic tool allowed for georeferenced intended behavior, improving measurement properties of the instrument. There were nonetheless obvious trade-offs on response error, coverage error and sampling error (Auld et al., 2012).

There are other options such as SMS, interactive voice response, redirected inbound call sampling, robocalls, and social media surveys that may afford researchers cost flexibility but also some trade-offs. Such surveys could, for instance, be perceived as informal, nonserious surveys, unwanted calls, illegal messages, or sales calls. The intelligence analyst needs to consider possible sources of error related to the mode of administration and the trade-offs in terms of total survey error.

ESTABLISHING CREDIBILITY OF CONTEXTUAL INFORMATION AND SOUNDNESS OF TECHNICAL DATA

A global interpretation of public opinion results is not always straightforward, but there are elements that can help to understand critical aspects about a public opinion study: information that is not typically included in reports and information that is expected to show up in survey methodology reports. Tables 3A-1 and 3A-2 provide a summary of helpful questions to collect and assess contextual and technical information of a survey. It is important to recognize that each factor in Tables 3A-1 and 3A-2 does not have equal weight, and analysts should consider some more than others. For example, when it is evident that a survey lacks authenticity (the survey is clearly a fake) it may be moot that the survey is based on nonprobability sampling methods or that there is limited coverage of the population. The questions are presented as guiding principles and as a shorthand method, but it is not by any means exhaustive. The intention is to give the analyst a star-based rating system to intuitively assess the survey based on scientific principles, and understand trade-offs around various factors that have an influence in the quality of results.

Once the analyst has collected as much intelligence as possible based on key questions, it needs to be sorted out and categorized. To ease the transition from information gathering to assessment, the analyst can think of sorting the contextual and technical information into a tiered system. All elements that receive low ratings (one to two stars), can be placed in a lower tier (“not credible, not sound”). Elements with three stars can be placed in a middle tier (“partially credible, partially sound”). Elements with four or five stars are then placed in a high tier (“credible, sound information”).

This page intentionally left blank.