Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop (2024)

Chapter: 5 Perspectives from the Small Business Community

Previous Chapter: 4 New Data and New Metrics for Evaluating Impact
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

5

Perspectives from the Small Business Community

Possible Metrics Discussed by the Presenters

  • The value of an innovation when integrated at the system level (Rozzi)
  • Spin-offs (Rozzi, Stocker)
  • Transitions of technologies to customers (Stocker)
  • Raising of private capital (Blatt)
  • Demonstrations of market competitiveness through direct revenues (procurement, commercial sales, licensing agreements) plus indirect revenues (spin-off sales, third-party sales from licensees) (Blatt, Rozzi, Carr)
  • Improved productivity that shows up in a company’s non-SBIR revenues (Blatt)
  • Increasing the size of the defense industrial base (Blatt)

On the second day of the workshop, the final two panels, which are summarized together in this chapter, turned to perspectives provided by representatives of the small business community.

A LACK OF DATA ON COMMERCIALIZATION

NAVSYS Corporation was founded in 1986 and established itself through an SBIR contract from the Air Force for a range-tracking telemetry system using GPS. It is in the top ten for companies receiving SBIR awards from the DOD in Colorado, has a high commercialization success rate for its SBIR projects, and has won prestigious state and national awards for its work. After almost four decades it still wins SBIR awards and remains a small company, and this small size works to its advantage, said Alison Brown, the company’s president and chief executive officer. “You can pivot, you can be supportive, you can be innovative, and the SBIR program allows all of that.”

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

Brown listed a number of challenges that small businesses have specifically in receiving Phase III awards. Burdensome contracting procedures make it difficult and slow for companies to receive Phase III awards. Language in the federal acquisition regulations (FAR) exempts research and development (R&D) contracts from normal “rule of two” small business contract set-asides, so small businesses that develop the prototype are having to compete against prime contractors for follow-on funding. Furthermore, although a success in the past, General Services Administration Phase III contracting processes are now overtasked and under resourced. Solicitations often demand government purpose rights and do not recognize that SBIR data rights are required to be nonnegotiable in Phase III awards. An increase in the use of controlled unclassified information in DOD SBIR topics creates restrictions that make it even more difficult for small businesses to obtain permission to release marketing information for commercial or government sales. “These types of burdens not only hurt companies but hurt their ability to commercialize technology,” Brown said.

A lack of data on Phase III SBIR funding prohibits accurate characterization of returns on investment for the DOD, she pointed out. Phase III awards are self-reported through Company Commercialization Reports, but only when applying for new SBIR awards, and they are unaudited. The government’s Federal Procurement Data System–New Generation database includes SBIR Phase III as a category for contract awards, but Phase III funding received through subcontracts, while common, is not reported as a category in the electronic subcontracting reporting system (eSRS.gov). The government has pivoted to using consortiums and other transaction authorities (OTAs), especially for prototypes, but Phase III funding for prototype development or continued research through OTAs is not reported. As a result, there is currently no reporting on funding provided to nontraditional contractors through OTAs.

The National Defense Authorization Act for Fiscal Year 2019 added incentive requirements in all DOD contracts of more than $100 million for use of SBIR/STTR technologies. In addition, while she said that DOD instruction 5000.02 included requirements for goals in leveraging SBIR/STTR funding and in adding incentives for primes, she said that these have rarely (if ever) been implemented by program officers.1 To meet this need, Brown said, incentives such as proposal evaluation criteria and incentive fees should be established for primes to insert SBIR Phase III technology into their contracts. In addition, SBIR Phase III awards should be reported in primes’ subcontracting plans and used for performance evaluations.

Consortia are not required to report on SBIR Phase III awards, Brown observed. Consortia-managed OTAs could speed SBIR/STTR Phase III transitions, but small businesses must pay a fee to a consortium to bid on awards, which is prohibitive because of the large number of consortia. Brown

___________________

1 DOD instruction 5000.02 establishes policy and prescribes procedures for managing acquisition programs for the Department of Defense. https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/500002p.pdf

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

recommended providing funding for SBIR/STTR companies to participate in these consortia and that consortia-managed OTAs be required to provide annual reports on Phase III funding. She also recommended that the DOD establish Phase III goals and include proposal evaluation factors and incentive award fees for meeting goals in all FAR or OTA awards made to companies other than traditional defense contractors.

In response to a question about OTAs, Brown cited the large number of innovation institutions within the DOD. “They are so fractured,” she said. They have good missions and are doing things that support their objectives. But they would be more effective if they were consolidated, she said, and if they focused their efforts on nontraditional contractors.

ACCOUNTING FOR DEMAND-SIDE SUCCESSES

Prime contractors and the DOD work in sync to use the SBIR program to fill technology needs for critical programs, explained Jay Rozzi, vice president and a principal engineer at Creare LLC. For new programs, they identify technology needs early in the relevant stage (development, design, production, operation, and sustainment) and develop technology roadmaps. For existing programs, the DOD does the same thing independently of the primes.

However, it takes time to develop these technologies. With adequate planning, the technologies can be ready at the right time for insertion, and “SBIR is a great way to do this,” said Rozzi. This can be an intensely competitive process at the Phase I and Phase II levels. From what can be dozens of Phase I proposals, two or three Phase I contractors compete usually for a single Phase II award. “Nowhere else in government are you going to see that level of competition for a single technology.”

SBIR and STTR “success” metrics focus on the small businesses—that is, on the supply side. However, Phase III (that is, non-SBIR) revenues can come from programs of record, private industry, or others. Direct revenue comes from product sales, license agreements, and other sources, while indirect or third-party revenue comes from product sales by licensees (minus license fees) or spin-off sales tied to the foundational SBIR technology.

“What’s missing is the demand-side focus,” said Rossi. In the case of the DOD, many projects are focused on increasing the lethality of the warfighter—“that’s what this is all about, getting the best solutions in their hands.” This may be in the form of reduced cost of production of a system, reduced operational constraints, or reduced cost of sustaining a system in the field. Before a technology is integrated into a production line, primes do a detailed calculation of the value of a technology over the lifetime of its integration, “and they don’t make that integration happen unless those numbers work.”

The value of an innovation when integrated at the system level is not captured currently, said Rozzi. For example, sometimes technology transitions occur during Phase II, but that is not accounted for anywhere. A large company can absorb that within the broader commercialization successes that it has had

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

over time. But a single two-person company getting involved in its first Phase II SBIR will not necessarily get credit for making a successful transition.

FLAWS IN THE SBIR DATABASE

John Stocker, who is retired from his position as senior vice president for federal solutions at Lynntech, Inc., agreed with Brown and Rozzi that SBIR contracting is very difficult. “We have complained bitterly to the Army, for example, about the fact that we’ve been selected for a Phase I award and it would take them a year, if not longer, to issue the contract. If you’re a small startup that’s very difficult to deal with.”

One problem with establishing benchmarks to identify success is that the database has many flaws. For example, the additional dollars awarded under Phase II contracts are not counted toward a success rate. “My position is that, if you have a program that’s willing to put dollars into a contract, that should be a recognition that you are successfully moving that technology forward.” Innovation is involved in Phase II as well as Phase I work. “Scientists can develop new innovative principles, but it takes an engineer to turn it into a product.”

More broadly, said Stocker, the acquisition system is a major impediment to innovation because technologies need to be identified that will be inserted into platforms 5–10 years from now, and the system does not allow for much flexibility when developing program objective memoranda, which establish early budget outlines for projects. In addition, the acquisition system discourages risk taking among program managers, making new technologies difficult to incorporate.

Because of the specialized demands of military technologies, fewer technologies than would be expected transition into the commercial sector. “It doesn’t make sense to talk about, in many cases, transitioning an SBIR-funded technology into the civilian market,” Stocker said. “There are obviously cases where that has occurred. But in the cases that we’ve been involved with, across a broad range of fields, even in the near-commercial market of DOD medical, it still is a problem because of what [DOD’s] requirements are.”

Stocker also advocated for a testing funding line in SBIR programs. “I can’t tell you the number of battery development projects that we’ve worked on, particularly with the Navy over the past few years, where there hasn’t been enough testing done. That’s because we don’t have enough money in our budgets to allow for that.”

Regarding metrics, gaps and shortfalls surround the gathering of data, he said. “It’s very difficult to explain to politicians why the database has problems and why you can’t make assumptions about what’s in that database and conclude whether someone’s a successful company or not.” In addition to a focus on companies, there needs to be a similar metric for customers to encourage transitions. Furthermore, penalizing companies that have won multiple SBIR awards is “the wrong way to approach the innovation development process,” he said.

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

FIRST PANEL DISCUSSION

During the discussion session, the members of the first panel commented on the effectiveness of the open topics solicitations used in recent years. Rozzi said that the fundamental question is whether open topics solicitations serve the interest of the DOD; “is the DOD getting value for its dollar?” There is a role for open topics within the lexicon of SBIR technologies, he said. However, although they were designed for new entrants, the structure of open topics grants nevertheless makes them difficult for new entrants to attain. They require obtaining a memorandum of understanding (MOU) from someone within the government saying that the technology will be transitioned if it is developed. “I understand the reasons for it. But if you’re a two-, five-, ten-person company, and you don’t know the market, the obstacles for you to do that are significant.”

Stocker asked whether open topics are more successful in transitioning technology than the traditional SBIR program. The open topics approach does not necessarily address the greater challenges of slow contracting, a difficult budget process, or funding that is susceptible to interruption by activities on Capitol Hill, he said.

Brown pushed back on the idea that open topics are new. The SBIR program has always had general topics that fit under strategic directions established by the department. In addition, the management of open topics solicitations has not always been sufficient to meet its objectives, she said. “Opening up for innovation for ideas is great, but it needs to be focused and aligned with the objectives for the overall Department of Defense’s funding.”

In response to a question about spin-off companies, all three panelists said that their companies have spun off companies focused on specific technologies. As Stocker said, “Not only do we see ourselves as an incubator and a developer of spin-offs, but we also—and we’ve made this clear to a lot of people in this town—are fully prepared to mentor smaller companies in learning how to navigate the SBIR process.”

Rozzi recounted, “None of us as owners or managers of the firm is smart enough to know what 5 years from now is going to be the best technology. So what we do is hire smart, entrepreneurial people, we bring them into the organization, and we give them an opportunity to go off and pursue their business and execute on their ideas. . . . It improves the vitality of the firm and offers more opportunities for cross-pollination.”

Brown, for her part, said that she is especially proud of the former employees of her company who have been hired by others.

MEETING NATIONAL DEFENSE NEEDS

What new entrants to the SBIR program want is 90 percent aligned with what the three speakers in the previous panel want for their programs, said Eric Blatt, executive director for the Alliance for Commercial Technology in Government, a trade industry association founded by startups and donors to

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

reduce barriers to federal procurement of innovative technologies. Blatt was the first of three speakers on the second small business panel. Blatt observed that everyone wants funding to go to the best technologies and those technologies, once their technical validity has been demonstrated, to be applied to warfighter requirements. “No one wants to build something and have it sit on a shelf; we want it to transition.”

However, the future of the battlefield will be much different than in the past, he observed. According to a recent report from the National Intelligence Council’s Strategic Futures Group, the future battlefield will be revolutionized by advancements in four broad areas:2

  • Connectivity, including sensors, big data and analytics, and networking
  • Lethality
  • Autonomy, including unmanned vehicles and swarming
  • Sustainability, including advanced manufacturing and affordable mass

All of these are critically important areas for national security, said Blatt. But problems associated with each of these areas tend to be addressed in traditional ways. Legacy defense contractors operating on cost-plus contracts lack the talent, organizational structures, and incentives to develop technologies—and particularly software—at a level that is competitive with the commercial market. Government data systems typically lag the commercial sector by 10 or more years. Government-funded technology development projects routinely run over budget and are obsolete on delivery. Such companies have trouble recruiting and retaining the best talent. “These issues will cause us to lose conflicts if they are not addressed,” said Blatt.

Blatt identified three purposes of the SBIR program:

  • Procurement and fielding of advanced capabilities that serve the warfighter
  • Attracting talent and resources to serve national security objectives
  • Satisfying specific research needs identified by program offices

In particular, he said, the SBIR program is uniquely valuable in its ability to entice extraordinarily talented founders with highly qualified technologies (often backed by significant private resources) to apply their time and resources to national security objectives. “What can we do to take their talent and . . . entice them to come work on national security requirements? If your goal is to get excellent talent that’s currently sitting at commercially oriented companies and persuade them to work on government requirements, the SBIR program is second to none.”

___________________

2 Strategic Futures Group, National Intelligence Council. 2021. The future of the battlefield. https://www.dni.gov/files/images/globalTrends/GT2040/NIC-2021-02493--Future-of-the-Battlefield-Unsourced--14May21.pdf

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

These three purposes create three metrics for success. First, with regard to the fielding of advanced capabilities that serve the warfighter, SBIR dollars invested in a company should translate to improved productivity that shows up in the company’s non-SBIR revenues, whether via government contracts or commercial sales, Blatt said. Also, companies that otherwise would not engage with the DOD should use the SBIR program to begin selling technologies to the DOD. In this way, the DOD gains improved capabilities while the company gains large enterprise customers and increased revenues. “How do we take things that have already been built and tweak them a little bit or further refine them and adapt them to specific warfighter requirements?”

Second, with regard to attracting talent and resources to serve national security objectives, the DOD correctly wants high-performing, well-qualified tech talent at young, often venture-backed, companies to work on its problems. Open topics have revolutionized the effectiveness of the SBIR program in this domain. Highly capable startups do not have to search for SBIR topics. They have commercial objectives, product roadmaps, venture capital in many cases, and boards of directors that keep them focused on their product roadmaps. “It’s easier for companies to know that the opportunity is there, it’s easier for them to capture that opportunity, and you wind up getting new applicants to the program.”

On this front, the committee should consider analyzing the qualifications of the companies entering the SBIR program, Blatt said, asking such questions as:

  • Does company leadership have prior business experience in their fields?
  • Have company leadership had prior successful transitions?
  • Has the company successfully raised private capital?
  • Has the company demonstrated market competitiveness by selling products?

“If we want to look at the success of the program, and the types of people that it’s bringing in, these are reasonable places to look.”

Finally, with respect to satisfying specific research needs, government technologists often select research topics that cannot be turned into successful businesses, Blatt noted. R&D service firms are great at R&D but not at commercialization. Nevertheless, DOD program offices have technical problems that need solutions, and small businesses are often more nimble and better able to recruit, retain, and apply R&D talent than large businesses. Success here should be measured by transitions: If customers are funding the R&D, are they purchasing what they pay to develop?

The number one challenge for the SBIR program is transition, Blatt concluded. The government is not good at buying emerging technologies. It is hindered by process impediments, cultural impediments, and training and skill impediments. His proposed solution is a small business commercial transition program. Such a program would provide permanent funding to the top 10 percent of SBIR technologies and fund them at 10 times the current level to get those

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

technologies into full production. A flexibility requirement could account for innovation surprises and provide the DOD with an administrative tool to purchase new capabilities. Enhanced training for acquisition personnel could provide for better use of existing solutions that can solve problems. “Let’s help the acquisition workforce professionals understand what it means to buy these types of solutions and what sort of licensing models can work,” he said. “We need to provide some support for those folks.”

PROVIDING VALUE TO END USERS

Sentenai is an artificial intelligence–enabled data fusion and analytics platform that dramatically reduces the complexity and time required to turn operational data into actionable, domain-specific insight, said its cofounder and chief executive officer Rohit Gupta. It uses unsupervised machine learning to examine individual data streams and decide on the optimal way to store and index that data. The data are then made available for customers using data science tools such as visualizations.

Realizing that defense could be a potential market, the company began to explore its options and saw the SBIR program as an opportunity. It has submitted applications to both open and traditional topics and has had success with those applications, being in the process of transitioning one of its Phase II projects at the time of the workshop. “We’re a company that would probably not have entered the defense space if not for open topics,” said Gupta. “It was an attractive way for us to say, ‘Hey, we have this technology. Let’s go find a customer and get that MOU signed.’” The company had multiple potential customers interested in working with it because of the capabilities its technology provided.

However, the company has also encountered challenges with the SBIR program. “We’ve often felt that we were not necessarily being evaluated purely on technical capability.” Rather, the company’s ability to write proposals often seemed more important. “I think more competition is great. I want to win on my merits. And I want other companies who are providing software to compete with me. But when the proposal rules require you to fill out specific sections, for someone who doesn’t spend all day selling into the defense world, that can often be hard and creates a barrier to entry.” The open topics approach makes the process more straightforward while maintaining its competitiveness. The company can say, “Here’s what our technology does, here’s the value it provides, here are case studies of different things we’ve done with different types of customers, and then it lets us be evaluated mostly on the merits.”

The bottom line is the value the company provides to end customers, Gupta said. Measuring the value of that hinges on such factors as whether the best technology available is being used and whether the best people are evaluating the solutions to problems. Applying such metrics “would help us improve our [assistance to] the warfighter a lot more quickly than in the past.”

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

Asked whether expansion into the defense markets could constrain a company’s private-sector activity, Gupta pointed out that his board monitors such decisions. “We’re not the first company, fortunately, to try to sell into the government. There are very successful venture-backed companies that have been able to commercialize in both. That helped provide the justification. At the end of the day, we want to solve problems, and if those problems are the same in the commercial and in the government world, great.”

CONTINUING SUPPORT

Vita Inclinata Technologies, Inc., specializes in stabilizing systems, particularly for rescue baskets suspended beneath aircraft, said the company’s chief executive officer, Caleb Carr. The company began in a garage and was awarded an SBIR award through an open topics competition, after which it won additional SBIR awards and a 2020 Tibbetts Award.

However, at the onset of the COVID-19 pandemic, the company was on the verge of being defunct. “The SBIR program is great for bringing companies in, but it’s absolutely horrible for supporting them internally,” Carr said. Also, because the DOD makes bulk purchases, revenues are difficult to forecast.

To navigate this, the company adopted a different approach. It established a strong presence in Washington, DC, and began advocating for litter stabilization systems appropriations. The company has continued to be very involved in conversations on Capitol Hill about SBIR companies.

Carr addressed the issue of open topics submissions, which he said has been hindered by appropriations problems. In addition, “the DOD ecosystem to transition an SBIR company to procurement is nonexistent. Unless a program manager or some kind of budget line has flexible funds, . . . you’re out of luck.”

The overall policy regarding the requirements and appropriations processes must change, he said, which will require sustaining conversations between the DOD and Capitol Hill.

SECOND PANEL DISCUSSION

Asked how they would recognize and measure the “quintessential SBIR success story,” Blatt described a success as a great technology that came to the SBIR program, was developed and advanced under the program, graduated from the program, and was sold extensively to customers.

Gupta said that a successful SBIR technology would affect warfighting. An additional measure of success would be commercial applications.

Carr said that a critical metric of success should be the survival rate of SBIR companies. “The whole purpose of the SBIR program is to build up the economic [strength] of the United States. . . . That’s how you get Congress to put more money into this program.”

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.

This page intentionally left blank.

Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 41
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 42
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 43
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 44
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 45
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 46
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 47
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 48
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 49
Suggested Citation: "5 Perspectives from the Small Business Community." National Academies of Sciences, Engineering, and Medicine. 2024. Data and Metrics for the DOD SBIR and STTR Programs: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/27984.
Page 50
Next Chapter: 6 Reflections on the Workshop
Subscribe to Email from the National Academies
Keep up with all of the activities, publications, and events by subscribing to free updates by email.