C

Stakeholder Survey Methods

To obtain stakeholder input, the Human Spaceflight Public and Stakeholder Opinion Panel conducted a survey of several key stakeholder groups. The survey was conducted by NORC at the University of Chicago. This appendix describes the methods used to conduct the survey.

SAMPLING FRAME

After initial informal exploratory discussions with a variety of experts and stakeholders, the panel, in consultation with the committee, developed a list of stakeholders customized to meet the needs of this project (Table C.1). To build a sampling frame for the survey, the panel identified leadership positions within each of the stakeholder groups of interest, and then identified the individuals occupying these positions. For example, in the case of the industry group, this included CEOs and Presidents of corporations that are members of the Aerospace Industries Association, Commercial Spaceflight Federation, and the American Institute of Aeronautics and Astronautics. In the case of space scientists, the sampling frame included members of the National Academy of Sciences, Institute of Medicine, and National Academy of Engineering, with an interest in space, as well as officers and Board Directors of professional associations, such as the American Astronautical Society and American Astronomical Society.

Because the committee’s charge was to make recommendations for a sustainable program spanning the next couple of decades, the views of the younger generation were particularly important to capture. To develop a sampling frame of young space scientists, the panel assembled lists of American Institute of Aeronautics and Astronautics early career and graduate student award winners, National Science Foundation postdoctoral fellowship winners, and NASA Aeronautics Graduate Scholarship Program winners from the past 3 years.

A description of the lists used to generate the sampling frames for each of the stakeholder groups is provided in Table C.1.

The approach described yielded an overall sampling frame of over 10,000 individuals. Within each of the groups, a systematic random sample was drawn, for an overall sample of 2,054 cases. Duplicate records within each group were removed before sampling. Those individuals who appeared in the final sample for more than one group were flagged, for the purposes of the analyses, as members of each of the groups they were sampled from, but they did not receive duplicate requests to complete the survey. Because “NASA’s stakeholders” are not a clearly defined population and because we selected the stakeholders using sampling frames that were reasonable and convenient rather than comprehensive, the results from this survey cannot be generalized to all stakeholders.

TABLE C.1 Lists Used to Generate Sampling Frame for the Stakeholder Survey

In addition, we have not made any attempt to weight the data to compensate for differences in the sizes of the various groups or overlaps in their composition. As a result, combining the responses from each group does not represent the universe of all of NASA’s stakeholders. Nevertheless, the sample from each frame was a probability sample and we believe this methodology provides a broader and more diverse perspective on stakeholder views than a nonprobability sample would.

SURVEY QUESTIONNAIRE

The objective of the stakeholder survey was to provide input to the committee on stakeholder views of human spaceflight, and specifically:

• The rationales traditionally provided for human spaceflight and any new or emerging rationales
• The importance of human spaceflight in the context of tradeoffs and alternatives
• The consequences of discontinuing NASA’s human spaceflight program
• The characteristics and goals of a worthwhile and feasible program for the near future (e.g., next two decades)

The survey instrument was developed by the panel in consultation with the committee. The goal was to develop standardized questions that could be both self-administered (via a paper questionnaire or web) and administered by a trained interviewer (via phone). The final questionnaire is included in Appendix D. To minimize response order effects (respondent’s potential tendency to favor answer options appearing toward the beginning of self-administered lists and toward the end of lists administered by phone), two versions of the questionnaire were produced, reversing the order of the response items in 3, 4, 5, and 11. Respondents were randomly assigned to receive either version A or version B of the questionnaire, consistently across administration modes. (The version included in Appendix D is version A of the mail questionnaire).

DATA COLLECTION

To carry out the data collection, the panel selected NORC at the University of Chicago. Data collection began on September 17, 2013, and concluded on November 6, 2013.

To maximize response rates, the survey was conducted in three administration modes—mail, web and telephone—all monitored by a case management system that tracked the status and outcomes of all sampled stakeholders across modes. This ensured that sample members were not contacted once they had completed the survey. It also controlled the number of emails received by sample members who had not completed the survey so as to limit unnecessary burden.

As a first step, a paper questionnaire was sent to all sample members where a mailing address was available. The questionnaire was accompanied by a cover letter signed by the Chair of the NRC, along with a postage paid envelope for returning the completed questionnaire.

Follow-up mailings were sent to cases where the only contact information after several rounds of locating attempts was a mailing address. In addition, repeat mailings were sent to a small number of sample members who requested another mailed questionnaire when contacted by a telephone interviewer. Near the end of data collection, reminder letters were mailed via FedEx to the remaining sample members who had not yet responded within the Industry and Young Space Scientist strata to help boost participation among these groups. These letters included the link to the web survey, individual PINs, and the toll-free number that sample members could use to complete the survey by phone.

Receipt control and data entry systems were used to input responses received via mail with 100 percent data entry verification.

Following the initial mailings and emails, all non-responding sample members with a located telephone number were contacted by phone. Telephone interviewers were selected and trained from the existing group employed by NORC’s telephone center in Chicago, IL. Prior to interviewing, all interviewers passed a certification test to demonstrate that they were well-versed in the project’s purpose and survey administration. A system was also implemented to receive inbound calls via a toll-free number.

A web link to this mode of the survey was sent to all sample members with a known email address. Sample members accessed the web survey using an individualized PIN assigned at random during the compilation of the sample. A link to the web survey was also included in the initial paper mailing, through regularly scheduled email blasts, and by request.

Telephoning began on October 1, 2013. Follow-up calls were made on a regular basis to all non-completers. Exemptions were made for those who refused to participate. NORC’s telephone center supervisors monitored interviewers throughout the data collection process.

The research staff continuously monitored results for signs of any complications with the data collection process. The research staff also monitored the progress of the survey on a daily basis, tallying the number of completed surveys by mode and by strata, as well as the total number of completed surveys. Careful monitoring of the transition of cases between modes of contact was critical throughout. Any cases where it was determined that the sample member lived outside the United States, was deceased or incapacitated/unable to complete the survey were considered not eligible.

Six survey questions requested open-ended responses that required coding by the research staff. The answers were coded into categories developed by the panel. Two coders independently coded 100 cases completed early in data collection and inter-coder reliability was calculated. The resulting kappa statistics are shown in Table C.2 and indicating the coding was adequately reliable. These cases were selected proportionally from the eight stakeholder groups. NORC survey managers and the coding supervisor debriefed the panel on this initial coding process. Then, upon the completion of data collection, all open-ended responses were coded in compliance with guidelines delineated during the initial batch of open-ends.

By the end of the data collection period 1,104 individuals, or 54 percent of the initial sample, completed the survey. The AAPOR Response Rate 3, which adjusts for ineligible cases, was 55.4 percent (Table C.3). In an effort to reduce the differences in response rates among the groups, at the end of the field period an additional mailing was sent via FedEx to nonresponders in the groups with the lowest response rates. To some extent, the differences

TABLE C.2 Inter-Coder Reliability Calculated Based on the First 100 Cases Coded

TABLE C.3 Number of Completed Cases and Response Rates

in the final response rates reflect differences in the contact information available for the different groups, including whether efforts to reach the sample member were likely to be screened by a gatekeeper, which was especially likely in the case of the industry group.