Summary

The National Science Foundation (NSF) asked the National Academies of Sciences, Engineering, and Medicine to conduct a quadrennial review of NSF’s Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs, in accordance with a legislative mandate. The committee convened by the National Academies to carry out this request conducted quantitative and qualitative analyses of data to produce this report on the operations of and outcomes stemming from NSF SBIR/STTR awards.

The report presents a detailed examination of the SBIR and STTR programs at NSF in accordance with the legislative mandate. Drawing on published research plus existing data, the committee analyzed (1) the effectiveness of NSF’s award selection process and postaward assistance; (2) the effectiveness of the STTR program in stimulating new collaborations; (3) the programs’ economic and noneconomic impacts; (4) the programs’ effectiveness in stimulating technological innovation and supporting small, new firms across the technological spectrum; and (5) the role of the programs in providing early capital to firms without other significant sources of support.

The committee’s lack of data on applicants impeded its ability to assess the efficacy of NSF’s current outreach effort in increasing the number of applications from small businesses that are (1) new to the SBIR/STTR programs, (2) from underrepresented states, (3) woman-owned, or (4) minority-owned. Those analyses were therefore limited in focus to the changing demographics of the awardee pool.

The committee found that NSF’s SBIR and STTR programs are funding young, small businesses that are new to the programs, as shown in Chapter 4. Analyses in Chapter 5 show that awardees have contributed scientific publications and patents and have realized financial and commercial outcomes that are consistent with the programs’ legislative goals. The programs also help support small, new firms across a wider technological spectrum than is supported by venture capital while also helping to diversify the geographic reach of seed funding, providing initial funding for companies in states, such as Michigan, Wisconsin, and Indiana, that receive little venture funding.

NSF has also increased the share of its SBIR/STTR awards going to woman- and minority-owned businesses, as described more fully in Chapters 4 and 5. The increase in awards to woman-owned businesses, especially Phase II awards, is significant. For minority-owned businesses, the results are mixed. Although Black- or African American–owned businesses have seen a slight increase in the share of awards, the share of awards to small businesses owned by Asian Americans has declined. Collaborations between small businesses and researchers at historically Black colleges and universities are rare. And the share of funding to Latine or Hispanic-owned businesses is less than is the case in the private sector. Woman- and minority-owned businesses that do versus those that do not participate in NSF’s early-stage technology commercialization support program, Innovation Corps (I-Corps), are more likely to win SBIR/STTR awards.

NSF has made a substantial number of changes to the two programs since their inception and has made the programs more accessible to new applicants, likely thanks to the recruitment of program directors with both technical expertise and industry experience. Some of NSF’s enhancements, such as requiring a short preapplication called the Project Pitch, are designed to give new applicants early feedback on potential applications. At the same time, many of NSF’s enhancements, such as the review of pitches, mean that program directors spend more time with new applicants and potential new applicants and have less time available to support awardees. Applicants that are new to the programs, especially those who have not previously submitted applications for any federal program, may have difficulty understanding what is needed to submit a successful application. Some of the difficulties faced by new applicants may be well suited to more generic assistance, such as an application assistance program, which would free up time for program directors and allow them to focus their energy and expertise on more technical issues facing awardees.

The centralization of the programs within one part of NSF lends itself to clear and uniform processes and procedures. At the same time, NSF’s rigidity regarding relatively small award amounts, as well as the limited opportunities for additional funding, may make the programs less attractive to firms working with capital-intensive technologies that are at an early stage of development. After receiving an NSF SBIR/STTR award, such firms move on to other federal agencies that offer SBIR and STTR programs so they can receive the early-stage funding they need to commercialize their innovations. Very few NSF grantees from outside of the SBIR/STTR programs have subsequently received SBIR/STTR awards, which may indicate a missed opportunity for NSF to facilitate the commercialization of the research it funds across the agency. The establishment of the new Directorate for Technology, Innovation and Partnerships (TIP) presents a unique opportunity for NSF to expand its diversity and inclusion efforts and to connect the SBIR/STTR programs with NSF-funded research in other directorates.

The lack of systematic data collection, data access, and preregistered evaluation approaches (i.e., one where the evaluation is planned and made public)

at NSF limited the committee’s ability to fully assess the agency’s programs. In particular, the committee was unable to determine which of a number of programmatic changes to the NSF SBIR and STTR programs have been effective at improving selection outcomes for woman- and minority-owned businesses, nor could it determine differences in commercial success between awardees and unsuccessful applicants. Granting access to data on all applicants for program assessment purposes, as is called for in the legislation mandating this review, and establishing processes that would allow for structured evaluation of policies and procedures would help NSF understand the effectiveness of its initiatives and how its programs could be improved.

From its complete set of 22 findings, the committee developed 8 recommendations, which are listed in full at the end of this summary. This summary presents an overview of the study and highlights the 14 findings and 4 recommendations the committee believes are most important.

SCOPE OF THE ASSESSMENT

Congress first requested that the National Academies undertake a study of the SBIR program as part of the Small Business Reauthorization Act of 2000 (expanded in the National Defense Authorization Act [NDAA] for Fiscal Year [FY] 2012), wherein Congress directed each agency that had an SBIR program budget of more than $50 million in fiscal year 1999 to engage with the National Academies to conduct a quadrennial assessment of its SBIR and STTR programs.¹ The congressional mandate calls for assessments that address how the SBIR and STTR programs have stimulated technological innovation, used small businesses to meet federal research and development (R&D) needs (SBIR), and stimulated technology transfer (STTR). The mandate also includes several specific analyses and evaluations, including the value and quality of R&D conducted under the programs and their economic and noneconomic benefits.²

STUDY METHODOLOGY, DEFINITIONS, AND LIMITATIONS

The committee based its review on a wide range of evidence, including descriptive and quantitative evidence regarding agency-level outcomes; quantitative data on patterns in the landscape of awards, firms, and related geographic and demographic characteristics; and qualitative evidence concerning the administration of the programs from the perspective of the agency, its

___________________

¹ Two previous full rounds of studies have been completed by the National Academies in response to the legislative mandate. The first was completed in 2009 and included a study of the SBIR programs at the Department of Defense, National Institutes of Health (NIH), Department of Energy (DOE), National Aeronautics and Space Administration, and NSF. The second, completed in 2016, included studies of both SBIR and STTR at the same agencies. More recently, the National Academies released a third DOE program assessment in 2020 and a third NIH program assessment in 2022.

² Section 108 of the Small Business Reauthorization Act of 2000 as amended by the FY2012 NDAA.

personnel, and awardees. The committee also used descriptive evidence regarding program impacts with respect to collaborations, firm structure/orientation, and other system-level outcomes that cannot easily be determined using standard econometric techniques. Finally, the committee considered causal evidence of direct and indirect effects of the programs on innovation and commercialization.

The committee gathered quantitative data from (1) the Small Business Administration’s (SBA’s) SBIR/STTR Company and Award Listing, (2) NSF’s database of awards, (3) SBA Dynamic Small Business Search, (4) the U.S. General Services Administration’s System for Award Management (SAM), (5) USASpending, (6) the U.S. Economic Development Administration’s Cluster Map, (7) the Federal Procurement Data System, (8) publication data in Web of Science, (9) patent data in PatentsView, (10) venture capital funding and initial public offering/acquisition outcome data in PitchBook, and (11) firm-level data in the National Establishment Time-Series database. Data from these sources were compiled, matched, and verified to provide the empirical basis for this study.

Because the committee was unable to access information on applicants who did not receive NSF SBIR/STTR awards, it matched a set of awardees with firms that won no awards based on a number of observable characteristics, such as firms’ age and size, as discussed fully in Chapter 5. While the committee made every effort to ensure the similarity of the counterfactual sample and the set of awardees, there are likely to be unobservable variables that differentiate the two groups. Therefore, it is difficult to determine to what extent the impact of receiving an NSF SBIR or STTR award is due to a selection effect (NSF’s ability to select the best applicants), the certification effect of being recognized as an NSF SBIR or STTR awardee, or the financial effect of the money associated with an award.

The committee also relied on qualitative data, including presentations by NSF SBIR/STTR personnel and consultations with current and former NSF personnel and researchers specializing in innovation. The committee consulted as well with a number of NSF SBIR and STTR awardees, I-Corps experts, and representatives of private-sector financing (including those from the venture capital, angel investment, and university accelerator communities). Finally, the committee used archival data available from the NSF SBIR/STTR website, such as webinars, publicly available documents, and funding opportunity announcements.

The committee uses the term “innovation” broadly to include activities up to precommercialization of new products and services, while also recognizing the three distinct stages in technological progress (invention, innovation, and diffusion of the innovation or commercialization) identified by Joseph Schumpeter. As described more fully in Chapter 2, the goals of the SBIR and STTR programs include meeting federal R&D needs, stimulating technological innovation, and increasing commercialization of innovations resulting from federal R&D funding—in other words, all three of these stages. Each of these

stages is difficult to measure; nonetheless, the committee endeavored to quantify each in its analyses.

Neither NSF nor SBA has a systematic dataset with uniform commercialization outcomes, which made the committee’s assessment of commercialization outcomes difficult. The committee was unable to obtain access to internal scoring for any of the SBIR/STTR applicants, nor did it gain access to information about applicants that were unsuccessful. The committee also lacked access to any demographic information about reviewers. These limitations impeded the committee’s ability to determine differences in commercial success between awardees and unsuccessful applicants or the root causes of changes in awardee diversity.

Finally, lack of access to the entire set of applicants limited the committee’s ability to analyze collaborations between small businesses and their research partners. Thus the committee’s analysis was limited to such collaborations within the STTR program.

KEY FINDINGS

As noted above, the committee developed 22 findings based on the available evidence, the most important of which are presented here. The first set of these key findings relates to economic and noneconomic outcomes from the agency’s SBIR/STTR programs.

Finding 5-1³: The NSF SBIR/STTR programs continue to play an important role in the U.S. innovation ecosystem. Notably, NSF SBIR/STTR awardees contribute scientific publications and patents and realize financial and commercial outcomes that are consistent with the programs’ goal of encouraging small businesses to engage in federal research and development with the aim of commercialization.

Finding 5-2: After receiving an NSF SBIR/STTR award, an awardee is more likely to advance science through published papers than is a similar business that does not receive an award. Over time, an NSF SBIR/STTR awardee is three times as likely to publish as a nonawardee. Prior to their receiving an award, publication rates for awardees were not statistically different from those for nonawardees.

Finding 5-3: NSF’s SBIR/STTR programs are effective at stimulating technological innovation. Although an NSF SBIR/STTR awardee is more likely than a similar small-business nonawardee to have produced patented inventions prior to receiving an award, it is eight times more likely to patent after receiving an award.

___________________

³ The committee’s findings and recommendations are numbered according to the chapter and the order in which they appear in the main text.

Finding 5-8: NSF’s SBIR/STTR programs are effective at selecting technologies that attract additional capital after an award has been received. After receiving an NSF SBIR/STTR award, an awardee firm versus a similar nonawardee firm is more likely to attract private-sector follow-on funding, to be acquired, or to complete an initial public offering. However, compared with a similar firm that attracts private investment instead of receiving an NSF SBIR/STTR award, an NSF SBIR/STTR–funded firm is more likely to fail.

As the findings above indicate, the committee found evidence that the NSF SBIR/STTR programs play an important role in the innovation ecosystem. These findings are based on the committee’s analysis of a limited set of NSF SBIR/STTR awardees compared with similar firms that have shown an interest in working with the government (by registering in SAM) but have not received an SBIR/STTR award. A higher failure rate for NSF SBIR/STTR awardees may indicate that NSF is selecting applicants whose innovations are higher risk and higher reward.

The next set of findings relates to the role of NSF’s SBIR/STTR programs in improving the diversity of start-ups. Compared with venture capital funding, the programs help provide early-stage funding for a broader set of technologies and broader geographic area. As stated earlier, NSF has increased the share of awards going to woman- and some minority-owned firms, although the results for minority-owned small businesses are mixed. For example, the share of awards going to Asian American founders has declined over the past decade. Collaborations between small businesses and historically Black colleges and universities are rare.

Finding 5-4: While software is still the most common industry sector represented by NSF’s SBIR/STTR awardees, the sectoral distribution of awardees is more diverse than that of firms supported by venture capital. NSF SBIR/STTR awardees that receive private-sector follow-on funding are more likely to be in pharmaceuticals, medical devices, semiconductors, chemicals, and other capital-intensive technologies than are firms funded by venture capital, which focuses on software and commercial services.

Finding 5-5: NSF’s SBIR/STTR programs fund start-ups in states where such firms receive little venture capital.

Finding 4-3: NSF has increased the share of its SBIR/STTR awards to woman- and minority-owned small businesses; however, the share of awards to small businesses owned by underrepresented minorities has

increased very little in the past decade, and the share of awards to small businesses owned by Asian Americans has declined.

Finding 4-7: NSF STTR recipients partner predominantly with universities, most commonly large, public, research-intensive universities. STTR recipients tend to use prior professional and educational networks when choosing a research partner. Historically Black colleges and universities are infrequent research partners for NSF’s STTR awardees.

NSF has experimented with and implemented new initiatives and has been successful at attracting and providing awards to applicants that are new to the SBIR/STTR programs and new to NSF. Although these initiatives are consistent with NSF’s broad mandate of advancing science, some (such as NSF’s restrictions on the number of applications and awards from a given small business and on award amounts) may not be ideal for other federal agencies. In that regard, NSF’s program appears to be a “feeder” for other agencies that may have specific capital-intensive technology procurement or research needs. However, the committee found that few NSF grantees from outside of the SBIR/STTR programs later received SBIR/STTR awards, which may indicate a missed opportunity for NSF to facilitate the commercialization of other NSF-supported research.

Finding 3-3: NSF has modified its SBIR/STTR programs over time as needed. It has instituted processes and procedures designed to increase the number of new applicants and to ease the administrative burden of applying, which is especially important for innovative small businesses.

Finding 3-2: A lack of flexibility in award size and limited opportunities for supplemental funding limit the attractiveness of the NSF SBIR/STTR programs for firms working with capital-intensive technologies.

Finding 4-4: NSF SBIR/STTR awardees are unlikely to receive subsequent awards from NSF. Many NSF SBIR/STTR awardees receive subsequent awards from other agencies, such as the Department of Defense and the National Institutes of Health.

Finding 4-5: SBIR/STTR awards can provide a route to commercializing research supported by NSF, but only a small share of NSF SBIR/STTR awards go to principal investigators with previous NSF (non-SBIR/STTR) awards. Most of the grants to prior NSF awardees are from NSF’s Engineering Directorate.

Finally, while the committee was able to assess NSF’s SBIR and STTR awardees compared with venture capital–backed business and with a set of similar businesses, more comprehensive data, including data on all applicants, would be necessary for a more comprehensive analysis.

Finding 5-9: NSF has not yet established systematic approaches to data collection, data access, and preregistered evaluation that would allow a comprehensive assessment of its SBIR/STTR processes and procedures.

Finding 5-10: NSF has made a number of programmatic changes to its SBIR/STTR programs, but it is difficult to determine which of these changes have been effective at increasing the number of awards to woman- or minority-owned businesses.

KEY RECOMMENDATIONS

On the basis of its findings, the committee formulated eight recommendations intended to help NSF and Congress continue to strengthen the NSF SBIR and STTR programs. Four of these recommendations stand out as key to improving the programs’ capacity to meet their stated missions and goals.

The first three of these key recommendations are geared toward NSF and its new TIP Directorate, which houses the NSF SBIR and STTR programs and NSF’s administration of the programs. First, to spur innovation in capital-intensive technologies that are often overlooked by venture capital, the committee recommends more flexible award amounts. The current legislative maximum amount for Phase II awards is nearly twice as high as what NSF allows. Additionally, given the dearth of applications from previous NSF grantees, the committee recommends that the TIP Directorate work more closely with the other directorates and encourage their awardees to consider applying for SBIR or STTR funding as part of an effort to commercialize their innovations. This would help NSF achieve the statutory goal of the programs of increasing private-sector commercialization of federal R&D.

Recommendation 3-4: NSF’s assistant director for technology, innovation and partnerships should give SBIR/STTR program directors greater flexibility in award amounts, including allowing for larger Phase II awards up to the statutory limit.

Recommendation 4-2: The new Directorate for Technology, Innovation and Partnerships should proactively encourage program directors across all NSF directorates to provide information on NSF’s SBIR/STTR programs to grantees conducting NSF-funded research and development.

Additionally, while NSF has made many changes to its SBIR and STTR programs, it would benefit from collecting evidence and establishing processes that would allow evaluation of the effectiveness of these policies and procedures.

Recommendation 5-2: NSF should establish systematic approaches for data collection, data access, and preregistered evaluation that would allow a comprehensive assessment of its SBIR/STTR processes and procedures. The data collected should include scientific, technical, and financial outcomes of awardees for 10 years following the end of an award. Data on firms that applied for but did not receive SBIR/STTR awards would provide a meaningful comparison group, and NSF should provide this information to the Small Business Administration as is legally required.

The committee’s final key recommendation is geared toward Congress. NSF’s SBIR and STTR programs have clearly demonstrated that they are meeting their legislative goals of stimulating technological research and of using small businesses to meet federal R&D needs. NSF has also made steady progress in fostering and encouraging the participation of women in technological innovation. To help NSF better meet its program goals, such as fostering and encouraging participation by members of minority and disadvantaged groups in technological innovation, Congress should continue to support NSF’s initiatives to improve its programs.

Recommendation 5-1: Congress should continue to support NSF’s SBIR/STTR programs, including by providing additional resources to enable innovation and new initiatives within the programs.

FINAL THOUGHTS

Over time, NSF has focused its SBIR and STTR programs on, and succeeded in, attracting new applicants to the programs. However, it offers limited subsequent funding opportunities for small businesses that receive awards. While these opportunities are consistent with NSF’s broad science mission, other federal agencies have targeted procurement or research needs that may require substantial investment before firms can attract private-sector funding. Congress designed the SBIR and STTR programs so as to allow federal agencies to tailor the programs to their specific mission needs. In service of fulfilling the programs’ legislative goals in the context of its own mission, each agency needs the latitude to pursue new initiatives and make programmatic decisions, such as adjusting award sizes and prescribing the appropriate maximum number of awards to any given small business. At the same time, NSF needs to adopt systematic approaches to data collection, data access, and preregistered evaluation to ensure that its new initiatives and programmatic efforts can be evaluated systematically, not only to

determine which of them should be discontinued, but also to determine which of them might be expanded or adapted in some way. Boxes S-1 and S-2, respectively, provide a complete listing of the committee’s findings and recommendations.

This page intentionally left blank.