An Assessment of the Small Business Innovation Research Program: Project Methodology (2004)
Chapter: Annex J Template for Individual Agency Reports
Annex J:
Template for Individual Agency Reports98
Introduction
Program history at the agency
Basic demographics of the agency program – number of awards, $ awarded, trends over past 20 years [drawing on program manager survey]
Organization – who runs what, under what offices, etc.
Short description – number of competitions annually, use of topics etc.
General methodology problems and approaches*
Comparability
Measurement
Bias
Multiple methodology approach, plus brief description of major instruments and approaches
Success stories box
Outcomes at the Agency
Commercialization Strategies and Outcomes
Difficulties of evaluating*
Research methodology
P2 survey data for the agency
Selected case study data for the agency
Agency-initiated analysis and data
Conclusions
Mission support
Short intro on agency differences, and procuring vs. non-procuring agencies*
Identification of specific agency as procuring/non-procuring
Research methodology
Agency interviews [within SBIR program and at more senior levels]
Procuring agencies only: Agency contracts analysis (DoD, NASA, parts of DoE)
Conclusions
Small business support
Problems of measuring additionality, and impact*
Research methodology*
If available, comparative impact study for specific agency (funded vs. unfounded applicants to SBIR)
Basic demographics of agency support for small business
Small business research $ as share of agency R&D spending; SBIR share of all agency R&D spending on small businesses
Conclusions
Knowledge base
Largely indirect character of knowledge expansion in the commercial environment; other “knowledge base” issues; comparability issues*
Research methodology*
Patents and trademarks
Licensing
Citation analysis
Equity sales [why is this here?]
Partnerships
Indirect path: indications, not measurement*
Conclusions
Program management
Topic development and selection
Sources for topic ideas
Agency-driven vs. investigator driven approach to topics
Topic decision-making
Outreach (before successful application)
Agency outreach objectives (e.g. reaching low-application states and regions; educating very likely applicants.)
Outreach programs that the agency runs or participates in (including brief description)
Agency support for P1 applicants
Agency outreach benchmarks and metrics
Grant selection
Description of selection processes for P1 and P2
Peer review panels – membership, selection, qualifications
Fairness – any issues and changes in procedures to meet problems; (P2 winner and non-winner views from surveys) [make sure there is a “fairness” question in the P2 survey]
Scoring procedures
Role of program manager
Re-submission procedures and outcomes
Training (after successful application)
Training programs for agency P1 and P2 grantees
Benchmarks used to evaluate effectiveness
Take-up rates and projections
Constraints
Phase 2+ program
Description of agency’s P2+ program
Use of matching funds
Application and selection procedures
Role of program manager
Funding “gaps.”
P1 – P2 gap. Average size, drivers. Options for addressing.
After P2. Data on subsequent agency funding/take-up of technologies. Agency perceptions of its role.
Bridge funding programs (after P2). Program description and objective Limitations (e.g. matching funding requirements)
What is the average extent of the gap
What is the primary cause of the gap
Does the agency think there is a problem
Is the agency finding ways to address the problem
Reporting requirements
Reports submitted to the agency by SBIR winners
Report utilization, and utility to the agency
Evaluation and assessment
Annual and intermittent agency evaluations of its SBIR program
Operational benchmarks for the agency SBIR program
Evaluators (internal and external)
Annual evaluation and assessment budget, and funding sources
Flexibility
Program managers discretion (project selection, project size, program management)
Program manager tools (e.g. shifting funding from one sub-agency to another; adjusting scoring criteria; changing the size of awards)
Agency manager perceptions of constraints
Size
Formal and effective limits on size and duration of awards
Distribution of P1 and P2 awards within those limits
Availability of additional funding. Amounts and distributions.
Online capabilities and plans
Components of the grant application, management, and reporting process NOT yet online
Plans and timelines for putting these remaining elements online
Barriers obstructing implementation of these plans
Administrative funding
Funding of program administration
Control of administration budget
Administration budget as percent of agency SBIR funding
Evaluation and assessment funding
Recommendations
Appendices
