Appendix B

Users of Science, Technology, and Innovation (STI) Data and Indicators and Their Questions and Requests for STI Indicators

USERS*

• Ana Aizcorbe (Virginia Tech, formerly at the Bureau of Economic Analysis)
• Shinichi Akaike (Hitotsubashi University)
• Jeff Alexander (SRI International)
• Michelle Alexopoulos (University of Toronto)
• Howard Alper (Canada’s Science Technology and Innovation Council)
• Rob Atkinson (Information Technology and Innovation Foundation)
• B.K. Atrostic, Cheryl Grim, Richard Hough, Dave Kinyon, Erika McEntarfer, and Mary Potter (U.S. Census Bureau)
• Asha Balikrishnan (Institute for Defense Analysis–Science, Technology Policy Institute)
• Carl Bergstrom (University of Washington)
• Stefano Bertuzzi, George Checko, and Jerry Sheehan (National Institutes of Health)
• Maria Borga (Bureau of Economic Analysis)
• Ray Bowen, Kelvin Froegemeier, Jose-Marie Griffiths, Arthur Reilly, and Arnold Stancell (National Science Board)
• Erik Brynjolfsson (Massachusetts Institute of Technology)
• Susan Butts (National Academy of Sciences, Government-University-Industry Research Roundtable)
• Jayanta Chatterjee (Indian Institute of Technology)
• Aaron Chatterji (Duke University, formerly of the Council of Economic Advisers)
• Cynthia Clark (National Agricultural Statistics Service)
• Patrick Clemins (American Association for the Advancement of Science)
• Alessandra Colecchia, Gili Greenberg, and Fernando Galindo-Rueda (OECD)
• Carol Corrado (The Conference Board)
• Gustavo Crespi (Inter-American Development Bank)
• Matthieu Delescluse (European Commission)
• Mark Doms (U.S. Department of Commerce)
• Nicholas Donofrio (IBM)
• Maryann Feldman (University of North Carolina)
• Changlin Gao (Chinese Academy of Science and Technology)
• Matthew Gerdin (U.S. Department of State)
• Lee Giles (Penn State University)
• Donna Ginther (University of Kansas)
• Martin Grueber (Battelle Foundation)
• Bronwyn Hall (University of California, Berkeley)
• John Haltiwanger (University of Maryland)
• Amber Hartman Scholz (President’s Council of Advisors on Science and Technology)
• Jonathan Haskel (Imperial College Business School)
• Christopher Hill (George Mason University)
• Hugo Hollanders (United Nations University-Maastricht Economic and Social Research Institute on Innovation and Technology)
• Matthew Hourihan (American Association for the Advancement of Science)
• Charles Hulten (University of Maryland, College Park)
• Adam Jaffe (Brandeis University)
• Tom Kalil and Kei Koizumi (U.S. Office of Science and Technology Policy)
• Bhavya Lal (Institute for Defense Analysis–Science, Technology Policy Institute)
• Julia Lane (American Institutes for Research)
• Brian MacAulay (National Endowment for Science, Technology and the Arts, United Kingdom)

____________________

^*All listed affiliations are as of February 2014.

• David McGranahan and Tim Wojan (U.S. Department of Agriculture)
• Daniel McGrath, Jessica Shedd, Matthew Soldner, and Tom Weko (National Center for Education Statistics)
• Christine Matthews (Congressional Research Service)
• Philippe Mawoko (The New Partnership for Africa’s Development)
• OECD-National Experts on Science and Technology Indicators (NESTI) workshop participants (30 member states, regional representatives, OECD staff)
• Richard Price (Academia.edu)
• Andrew Reamer (George Washington University)
• Alicia Robb (Kauffman Foundation)
• Carol Robbins (Bureau of Economic Analysis)
• Laurie Salmon, Jim Spletzer, and David Talan (Bureau of Labor Statistics)
• Robert Samors and David Winwood (Association of Public and Land-grant Universities)
• Walter Schaffer (National Institutes of Health)
• Jerry Sheehan (National Institutes of Health)
• Stephanie Shipp (Institute for Defense Analysis-Science, Technology Policy Institute)
• Dahlia Sokolov (U.S. House of Representatives)
• Gregoy Tassey (National Institute of Standards and Technology)
• Katherine Wallman, Rochelle Martinez, and colleagues (Office of Management and Budget)

KEY ISSUES AND QUESTIONS FOR STI INDICATORS

• Growth, competitiveness, and jobs: What is the contribution of science, technology, and innovation (STI) activity to productivity, employment, and growth? What is the relative importance of technological innovation and nontechnological innovation for economic growth? What are the current advances and vulnerabilities in the global STI system? Is the United States falling behind with respect to innovation, and what are the effects on socioeconomic outcomes? Where is leadership in science and technology (S&T) trending?
• STI activities: What are the drivers of innovation? How important are the following for advancing innovation: small businesses, large businesses, strategic alliances, technology transfer between universities and firms, academic researchers, government laboratories and procurement activities, and nonprofit organizations? How influential is research and development (R&D) for innovation and growth (by sector)? What is the role of intangibles in affecting productivity? How do government investments in S&T contribute to innovation? What is the return on investment in basic research, and what good does it do for society? What would constitute a “balance” between the biological and physical sciences? On what basis could that be determined? Does biological science depend on physical science for advancement? What are the emerging innovative sectors, and what is unique about them? What is the international connectivity of science, technology, engineering, and mathematics (STEM) activities?
• STI talent: What is the status of STEM education around the world? How much knowledge capital does the United States have? How many people, possessing what kinds of skills, are needed to achieve a robust STI system? What additional sources of “talent” can best be tapped—especially among immigrants, women, and minorities? What are the career paths of foreign-born STEM-educated or foreign-trained individuals? What fraction of STEM-degree holders have STEM jobs? What is the return on investment for individuals who obtain STEM education or training? How many science and engineering (S&E) doctorate holders took nontraditional pathways into the STEM workforce? Did this vary by race/ethnicity, gender, or the existence of a disability? How important are community colleges in developing human resources for STEM talent? What is the trend in online learning and massive open online courses (MOOCs) in the United States and abroad? Is the United States falling behind in STEM workers? What fields other than STEM are important for advances in STI? What are the fields that contribute expertise to advances in clean energy?
• Private investment, government investment and procurement: What impact does federal research spending have on innovation and economic health, and over what time frame? How large should the federal research budget be? How should policy makers decide where to put additional research dollars or reallocate existing funding streams—information and communication technology (ICT), biotechnology, physical science, nanotechnology, environmental technology, social science, etc.? Does government investment crowd out or energize private investment in STI activities? What is the role of entrepreneurship in driving innovation?
• Institutions, networks, and regulations: What impacts are federal research programs (including federally funded research and development centers) having on entrepreneurial activities in S&E sectors? Where are the key gaps in the transfer of scientific and technological knowledge that undercut the performance of the STI system? Where is the supposed “valley of death” in innovation? In which industries is the valley of death most prevalent? What part of the process is underfunded for specific sectors? What is the nature and impact of intellectual property

• protection with respect to scientific and innovation outputs? How do incentives for innovation activities work (national and state levels)?
• Global STI activities and outcomes: What can the United States learn from other countries, and what are other countries learning from the United States? In what technological areas are other countries accelerating? What impact does the international flow of STI have on U.S. economic performance? What is the relative cost of innovation inputs in the United States versus other countries? Where are multinational corporations sourcing R&D? What are the institutional differences that affect innovation activities among nations, and how are they changing?
• Subnational STI activities and outcomes: How does innovation activity in a given firm in a given place contribute to that firm’s productivity, employment, and growth, and perhaps also to these characteristics in the surrounding area? How are those innovation supply chains working within a state? Are firms outsourcing new knowledge principally from customers or from universities?
• Systemic changes on the horizon: How is the global STI ecosystem changing or evolving? What sectors, regions, and people will rise in prominence in the near future? How will demographic shifts affect the STEM workforce, nationally and internationally? Will they alter the locus of the most highly productive regions? Will global financial crises slow innovation activities or merely change the locus of activities? When will emerging economies be integrated into the global ecosystem of innovation, and what impact will that have on the system? What changes are expected in the following sectors: clean energy, agriculture, biotechnology, nanotechnology, information technology, cyber technology, weapons, “big data,” and others? How are public views of S&T changing over time? What is the culture of innovation (e.g., entrepreneurship, willingness to take risks) in U.S. regions and around the world? What are the general perceptions about science and the public value of science in the general population (United States and abroad)?

KEY INDICATORS SUGGESTED BY MAJOR USERS OF STI INDICATORS

Activities

R&D

• National R&D expenditures

-   Federal and state funds for basic research

-   Public-sector R&D (focus on advanced manufacturing, green technologies, energy-related R&D, nanotechnology, agriculture, weapons)

-   Public R&D spending as a share of gross domestic product (GDP)

-   Business R&D spending

-   Business R&D as a share of GDP

-   Industry support for R&D in universities

-   Social science R&D

• National R&D performance (by type of industry and source of funds)
• Trends in size of grants to universities
• Number of R&D centers in the United States and other countries

Innovation

• Direct measures of innovation (Community Innovation Survey-like data)

-   Propensity-to-innovate ratings

-   Subject matter experts (SMEs) innovating in house as a share of SMEs

-   Firms (<5, 5+, 10+, 20+ employees) introducing new or significantly improved products or processes as a share of all firms

-   Firms (<5, 5+, 10+, 20+ employees) introducing new or significantly improved goods or services as a share of all firms

-   Firms (<5, 5+, 10+, 20+ employees) introducing marketing or organizational innovations as a share of all firms

• Number and types of new products per year, by region (Thomasnet.com)
• Drug and other approvals per year, by region
• Sale of new-to-market and new-to-firm innovations as a share of turnover
• Non-R&D expenditures on innovation activities; non-R&D innovation spending as a share of turnover
• Inclusive innovation for development (case studies)
• Capital expenditures related to the introduction of new processes
• Marketing expenditures related to new products
• Expenditures on design and technical specifications
• Expenditures on service-sector innovation; investment in new ICT hardware and software
• Innovation inhibitors (case studies)

Market Capital Investments

• Venture capital investments in S&T (early-stage, expansion, and replacement); venture capital in S&T as a share of GDP
• Number of initial public offerings in S&T
• Number of S&T spinoffs

• Expenditures in later phases of development/testing that are not included in R&D

Outputs and Outcomes

Commercial Outputs and Outcomes

• Performance of “gazelles,” small firms and small business units within large firms
• High-growth enterprises as a share of all enterprises
• Medium- and high--tech manufacturing exports as a share of total product exports
• Exports of knowledge-intensive services as a share of total service exports
• Value added in manufacturing
• Value added in technical services
• Trade flows of S&T products and services
• ICT output and sales (intermediate and final)
• Other intermediate inputs
• Technology balance of trade (especially intellectual property)
• Contracts to S&T firms
• Advanced manufacturing outputs (information technology-based processes)
• Market diffusion activities
• Emerging industries (based on universities, government laboratories, firms, value chains, key occupations, and individuals)
• Business practice data (e.g., help-wanted ads, “how to” books)

Knowledge Outputs

• U.S. receipts and royalty payments from foreign affiliates
• U.S. patent applications and grants by country, technology
• U.S. trademark applications and grants by country, technology
• Patent citations
• License and patent revenues from abroad as a share of GDP
• Triadic Patent Families by country
• Percentage of patent applications per billion GDP
• Percentage of patent applications related to societal challenges (e.g., climate change mitigation, health) per billion GDP
• Intangible assets
• Average length of a firm’s products’ life cycles or how often the firm usually introduces innovations
• Births and deaths of businesses linked to innovation outputs; firm dynamics by geography, industry, business size, and business age
• Knowledge depreciation
• Knowledge stocks and flows in specific sectors, including nanotechnology; information technology; biotechnology and agriculture research (local foods, organic foods, biofuels, environment, nutrition, health); oil and gas production; clean/green energy; space applications; weapons; health care technologies; educational technologies (MOOCs); mining

STEM Education

• Expenditures, direct and indirect costs, investments, revenues, financing on STEM education
• Percentage of faculty in nonteaching and nonresearch roles at universities
• Enrollment data by STEM at various levels (e.g., associate’s, bachelor’s, master’s, doctoral degrees) and for various types of institutions
• New degrees (e.g., associate’s, bachelor’s, master’s, doctoral); new doctoral graduates per 1,000 population aged 25-34
• Stock of degrees (e.g., associate’s, bachelor’s, master’s, doctoral)
• Share of population aged 30-34 having completed tertiary education
• Share of youth aged 20-24 having attained at least upper-secondary-level education
• Persistence and dropout rates in education by geographic and demographic distinctions
• Number of high school students pursuing associate’s degrees and implications for the workforce and the cost of higher education
• Disciplines in which community colleges have a comparative advantage
• Foreign-born STEM-educated individuals—countries of birth, immigration visas, etc.
• Stay rates of foreign students
• Trends in online learning and MOOCs

STEM Workforce/Talent

• Postdoctoral levels and trends in various STEM fields by country of birth and country of highest degree
• Number of postdoctorates in health, specific fields
• STEM employment
• Labor mobility and workforce migration
• Demographic composition of people who would enter specific occupations (e.g., clean energy, ICT, biotechnology, health services)
• Fraction of STEM degree holders that hold STEM jobs
• Earnings by degree type and occupation
• Feeder fields in agricultural science
• On-the-job training activities in S&T, manufacturing, and services
• STEM demand

• Employment in knowledge-intensive activities (manufacturing and services) as a share of total employment

Socioeconomic Impacts/Well-Being (The “Are We Better Off” Question)

• Economic growth
• Productivity
• Other measures of impact on GDP and jobs
• Agricultural preparedness
• Energy preparedness
• Return on investment on grants to universities by type of S&T
• National security/defense
• Environment
• Energy use
• Geographic hot spots

Linkages

Organizations/Institutions

• Public-private copublications per million population
• University-industry research collaborations
• Number and value of international collaborations
• Business structure dynamics
• Technology transfer between academic institutions and businesses, including mechanisms
• Technology transfer (including programs such as Manufacturing Extension Partnership Technology Transfer/Transition Pilot Initiative)
• Technology transfer from national laboratories
• Bilateral S&T agreements (including international)
• Collaboratories
• Industry clusters
• Incubators
• Consortia (Defense Advanced Research Projects Agency [DARPA], Advanced Research Projects Agency-Energy [ARPA-E], Technology Innovation Program at the National Institute for Standards and Technology)
• Intellectual property rights and policies
• Standards
• Market planning assistance (Department of Commerce [DoC], Bureau of Labor Statistics [BLS], Small Business Administration [SBA])
• Research and experimentation (R&E) tax credits (federal and state)
• Innovative SMEs collaborating with others as a share of SMEs
• Alumni contributions to R&D

Culture

• Public value of S&T
• Business climate
• Entrepreneurial activities

-   Mappings of entrepreneurial density

-   All establishments and firms with at least one employee, including start-ups, from 1976 to the present

-   All nonemployer firms and integrated-with-employer firms from 1994 to the present

-   All employer-employee matches and transitions (hires, separations, job creation, and job destruction) from 1990 to the present

-   Information on innovation policies and outcomes (contexts; national, regional, sectoral levels)

-   Data on existence of dealmakers and entrepreneurs and their connections in a given market

• Risk tolerance
• Social networks

-   Attendance at conferences (number of times per year)

-   Membership in associations (number of associations linked directly and indirectly to field of study or occupation)

-   Linkages to sources of capital

-   Linkages to sources of knowledge or ingenuity used in occupation

Subnational Indicators

• State, county, and metropolitan tables of data from the Business Research and Development and Innovation Survey (BRDIS) (covering R&D performance, workforce, and intellectual property)
• Academic R&D expenditures
• Federal R&D expenditures
• Industry support for R&D in universities
• Total R&D (from a resurrected nonprofit R&D survey)
• Degrees granted in STEM (production and migration)
• STEM jobs (Occupational Employment Statistics from BLS)
• STEM workforce migration (data on Local Employment Dynamics from the U.S. Census Bureau)
• Patent applications, grants, and citations (from the U.S. Patent and Trademark Office)
• STI equity investments (from various sources, including venture capital)
• STEM occupational projections (from BLS and the Employment and Training Administration [ETA])
• STEM occupation classification (from ETA)
• STEM graduate and workforce migration (National Center for Education Statistics, from the Census Bureau and BLS)

• Firm innovation processes (from the Economic Research Service [ERS] at the U.S. Department of Agriculture [USDA])
• Propensity-to-innovate ratings
• Mappings of entrepreneurial density
• Firm births, mergers and acquisitions, deaths (“business dynamics” as characterized by Haltiwanger at the panel’s July 2011 workshop, including geography, industry, business size, business age)
• State and federal grants and loans (from Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science [STAR METRICS])
• Initial public offerings
• New products (from Thomasnet.com)
• Drug and other approvals (from the Food and Drug Administration)
• Data on dealmakers and entrepreneurs, including number of connections
• Data on emerging industries, based on universities, government laboratories, firms, value chains, key occupations, and individuals