6

Reflections on the Workshop

The workshop agenda included opportunities for reflections on the proceedings at the end of each day. These reflections are collected in this chapter.

DAY 1 REFLECTIONS

At the end of the opening day of the workshop, coming before the final two panels on the experiences of small businesses with SBIR programs (summarized in Chapter 6), members of the committee reflected on the messages they heard emerging from the workshop’s presentations and discussions. Their remarks do not necessarily represent the views of the committee or the committee’s internal deliberations, said committee cochair Scott Stern, but they are among the issues the committee will be considering as it prepares its report.

Ellen Lord emphasized that SBIR awards form part of a much broader innovation ecosystem within the DOD. That ecosystem includes the Defense Innovation Unit, rapid capabilities offices, and many other programs. “SBIRs are not meant to solve every problem out there,” she said. Also, the innovation ecosystem and national defense are regularly perturbed by new emerging technologies, new threat vectors, and other developments. Nevertheless, SBIR programs need to remain “laser focused on that Venn diagram intersection of the two key things: fostering the growth of small business while addressing defense-related challenges.”

Arun Seraphin made the same point—that SBIR is just one piece of a larger innovation ecosystem that the DOD is seeking to use and support. There are many metrics that can be applied to that system, and these metrics are important, in that DOD leadership is focusing attention on the program. Seraphin also pointed to the importance of examining the selection criteria that the DOD is using in the program, which are a product of law, policy directives, and the broader culture. Are these source selection criteria affecting the program in a positive or negative way, and can that be used to establish a metric for the program?

Seraphin said that small businesses need to form connections to undertake larger initiatives, whether the modernization priorities of the DOD, specific technology initiatives, or accepting the input of “good engineers from a prime who have a desire to help you incorporate your technology.” A “metric of positive management” could put a measure on this feature of companies.

Metrics could also evaluate the effectiveness of how funds flow within the system and to SBIR companies. Are funds flowing in such a way that the goals of the program are being achieved? In this respect, the flow of funds within SBIR programs is actually more flexible and shorter than the usual 2-year process in much of the rest of the DOD, which in theory could make it faster to respond to emerging threats or emerging technologies.

Finally, Seraphin noted that the DOD spends more money outside the SBIR program on R&D projects with small businesses than inside the SBIR program. Looking at the outcomes of this spending would provide good points of comparison with SBIR awards.

Kyle Myers observed that companies have incentives to report results in different ways, particularly when those ways are being used as evaluative measures. The committee therefore needs a way to look beyond these incentives to the actual results.

He also cautioned against simple measures of spending as a metric of success. In some cases, reduced spending might be a more appropriate measure. “Generating new metrics that are closer to the national security value, or the defense value, of the technologies created by this program will be a challenge to us.”

Maryann Feldman pointed out that something missing from the SBIR program is someone who can help small firms proceed toward commercialization. In addition, many new requirements and regulations have accrued to the program over its existence. “For us, understanding how the program is managed, how it is implemented, and the types of incentives that it offers are going to be very important.” She also pointed out that it is a 40-year-old program that still operates from some assumptions grounded in past realities. For example, the terminology of Phase I, Phase II, and Phase III awards implies a linear progression, but in fact innovation can proceed in other directions, as when Phase III projects reveal good ideas for Phase II initiatives. She also pointed out that the program was based on the premise that small businesses need to be brought up to speed and familiarized with the needs of the government. But today, especially with open topics, companies are often the leaders in establishing technological capabilities, at least in some areas.

Stern suggested that four important metrics discussed throughout the workshop are cost, risk, schedule, and performance. In some technological sectors and industries, it may make sense to emphasize one of these four at the expense of the other three. But within the context of the DOD’s SBIR programs, “it has to work within the system.”

Lord observed that Congress acts essentially as a board of directors for the DOD. “There is an enormous amount of back and forth, because it’s

Congress’s job to be good custodians of the taxpayer dollar.” The diversity of ideas and thinking within both Congress and the DOD produces considerable debate, not to mention the additional perspectives imposed by organizations like the National Security Council or the Office of Management and Budget. It also reinforces the need to meet all four metrics above.

A discussion about companies that win multiple awards did not yield a straightforward conclusion about possible metrics. One important procedural step, said Lord, is to identify in a request for proposals the selection criteria that will be applied to those proposals.

Finally, Lord remarked that innovative technology is not useful to the DOD unless it can be applied to warfighting. “It’s how you take that technology and package it into hardware, software, services, whatever it might be that meets a need.”

FINAL THOUGHTS

Feldman provided final wrap-up comments at the end of the workshop. She highlighted that the SBIR/STTR programs face a number of different programmatic goals, some of which may be inherently in conflict. “The various constituent groups have differing opinions about the programs’ goals, and this leads to differing expectations of what the programs should accomplish and therefore the metrics that should be used to evaluate the programs.” She highlighted the points made by Bhaven Sampat, who found that in the National Academies’ review of the National Institutes of Health’s programs, no one indicator or set of indicators satisfies the differing expectations for the program. Therefore, she said, data need to be collected and curated for a diverse set of metrics useful to improving the programs’ performance along multiple dimensions.

In addition, she noted that several of the speakers highlighted the fact that innovation doesn’t follow the simplified linear model, and one should not expect that every Phase I award will lead to a Phase II award and then a subsequent commercial outcome. But perhaps more important than the set of possible metrics, she concluded, is how many of this workshop’s speakers noted that the data needed to evaluate these programs are not collected in a usable fashion, even by DOD. “Many of the workshop’s participants have highlighted issues in underreporting the impact of SBIR/STTR awards on subsequent DOD procurement.” She continued that some of the underreporting may be due to the role of the awardees as subcontractors, rather than direct suppliers to DOD, but she noted that other speakers mentioned issues in the lag times between awards and impact (which may be over a decade), as well as “invisible” innovation if the application is related to national security concerns.

This page intentionally left blank.