An Assessment of the Small Business Innovation Research Program: Project Methodology (2004)
Chapter: Annex F Research Matrix
ANNEX F:
Research Matrix
SBIR Review Objectives | ||||||||
| Assessment of the program | Consideration of operational improvements to the program | ||||||
| quality of research | commercialization of SBIR funded research/economic and non-economic benefits | small business innovation/growth (incl.minority owned) | use of small business research to advance agency missions | program design | service to participants | efficiency of SBIR program administration | program evaluation |
Questions | * How does the qualityof SBIR funded research compare that of other government funded R&D? | * What is the over all economic impact of SBIR funded research? * What fraction of that impact is attributable to SBIR funding? | * How to broaden participation and replenishing contractors? * Possibility of link SBIR with state/regional programs? | * How to increase agency up take while continuing to support high-risk research? | * Time and award limits? * Review process and criteria? * Intra- and inter-agency interface? * New funding model to build collaborative networks? | * How can SBIR better serve technology entrepreneurs? * Lessons from angels? Corporate R&D? | * How to leverage economies of scale in program administration while allowing each agency full management autonomy? | * How to evaluate SBIR success: Appropriate measures? Impact of skew in returns? |
Measures | peer review scores, publication counts, citation analyses | sales; follow-up funding (e.g. angel, VC, corporate); drug development progress; IPOs | patent counts and other IP/employment or job growth, number of new technology firms | agency procurement of products resulting from SBIR work | ? | ? | speed of response to applicants; administrative cost per dollar awarded | [all] |
Key researchissues /challenges | * Difficulty both of measuring quality and of identifying proper reference group | * Skew of returns: estimates of returns invariably dominated by a few big "winners"` * Significant inter-agency and inter-industry differences | * Measures of actual success failure (both project and firm level) * Relationship of Federal and state programs in this context | Major inter-agencies differences in use of SBIR to meet agency missions | Documenting/addressing the problem of "backwater" status of SBIR programs within agencies | * Response time * Ease of application * Outreach activities | * Shift from paper to electronic program management * Possibility of using some funds for program administration | * Appropriate implementation of GPRA in this context? |
Desired outcomes | * SBIR isworking better* Projects are beingtracked* Measures exist of long-term impacts | |||||||
| Assessment of the program | Consideration of operational improvements to the program | ||||||
| quality of research | commercialization of SBIR funded research/economic and non-economic benefits | small business innovation/growth (incl. minority owned) | use of small business research to advance agency missions | program design | service to participants | efficiency of SBIR program administration | program evaluation |
Phase I survey |
|
| X |
| X | X |
| X |
Phase II survey |
| X | X |
| X | X |
| X |
survey of "program managers" |
| X |
| X | X |
| X | X |
case studies | X | X | X | X |
| X |
| X |
agency program studies | X |
|
| X | X | X | X | X |
proposed study of repeat winners | X | X | X | X |
| X | X | X |
proposed bibliometric analysis comparing SBIR firms with (similar) non-SBIR firms | X |
| X |
|
|
|
| X |
