Ellen Williams, Director

Advanced Research Projects Agency-Energy

U.S. Department of Energy Washington, D.C.

Pradeep Khosla

Chair of the Committee for the Evaluation of ARPA-E

Dear Director Williams:

Thank you for providing the National Academies of Sciences, Engineering, and Medicine Committee for the Evaluation of ARPA-E with the important information contained in your initial presentations and in the separately provided documents. The Committee appreciates all of the effort and care that you and your staff have already put into this project.

As stated during its last meeting in July 2015, the Committee requires additional information and data in order to fulfill its mandate from Congress to conduct an independent evaluation of the appropriateness and effectiveness of ARPA-E’s structure to achieve its mission and the most significant technical accomplishments of the program. We would like to accept your offer to provide additional information and data.

Some of the information the Committee needs includes program and project descriptions that you have already provided or is available through the ARPA-E website. I wish to verify that the Committee has the complete set of information. The Committee and I also recognize that you and your staff are working to complete an internal self-appraisal of the accomplishments and impacts of ARPA-E and plan to provide a presentation at our next meeting in October 2015.

Listed below is the Committee’s initial request for additional information. The committee may wish to obtain additional information through the course of its work. In obtaining needed information and data, the Committee and I wish to work with ARPA-E in any way we can – such as by utilizing independent consultants or NAS staff – to minimize the burden on your agency’s limited resources while still maintaining our independence. We also want to ensure that there is no compromise of ARPA-E’s necessary confidentiality or other legal obligations while fulfilling the Academies’ Congressional mandate to conduct an external evaluation. I note that many of these information sources were made available for the Government Accountability Office’s review of ARPA-E that was completed in 2012.

The Committee would like to know how ARPA-E:

• Defines ‘success’ and ‘failure’ for both programs and projects; and
• Captures and propagates lessons learned from program or project ‘successes’ and ‘failures’ institutionally and across program directors and, where appropriate, to other offices and programs within the Department of Energy.

The Committee respectfully requests:

• Description of Project Performers – we would like to verify that we have complete descriptions on the full portfolio of both open and focused programs and projects, including information to distinguish whether this is a seedling research effort intended to help develop new focused programs or a “regular” project intended to, eventually, produce a specific commercializable device, the description and location of the performer, and the funding level;
• Access to ARPA-E internal program documents for all focus programs and broad area announcements to date including official Funding Opportunity Announcements, submitted projects, and selection processes;
• For each full application received, detailed project and PI information (e.g., lead-PI, Co-PI and other associated staff and personnel) whether the project was funded or not, including information used to make a fund/don’t-fund decision (such as applicant scores);
• For each project funded, a detailed description of milestones and metrics and individual projects’ performance against those milestones and metrics;
• A list of instances where project milestones were revised during the course of the project, including descriptions of why ARPA-E and the performer changed the milestones and how those changes impacted the project;
• A list of projects restructured, particularly in response to revising milestones;
• A list of projects terminated, including descriptions of why and how ARPA-E and the performers terminated the projects; and
• A description of the process for terminating or restructuring, including:
  • An explanation of how ARPA-E decides it must discuss restructuring or termination with a project performer; and
  • Whether and how input from outside experts (at dedicated ARPA-E focus program events or the ARPA-E summit, for example) is utilized when making a decision to restructure or terminate a project.

The Committee respectfully requests:

• Copies of final project reports for all completed projects;
• Lists of journal and other scientific publications derived from ARPA-E funded projects;
• Lists of patents that derive from ARPA-E funded projects;
• Data on commercialized outcomes, revenues generated, and follow-on funding acquired by ARPA-E grantees/awardees;
• Data tracked by ARPA-E on performers following completion of individual projects; and
• Graduate students and postdocs funded by ARPA-E programs, and their career trajectories (e.g., academics, industries, startups, or government).

The Committee is interested in ARPA-E’s work in developing new communities and respectfully requests:

• ARPA-E’s description of “new communities” that have been built or enhanced as a result of ARPA-E programs, and evidence to support ARPA-E’s role and contribution in forming those communities;
• A description of how these new communities support or have contributed the goals of ARPA-E;
• Programs and agendas from past and planned ARPA-E pre-program workshops, program kickoff meetings, program annual meetings, and annual innovation summits;
• Lists of attendees present at ARPA-E pre-program workshops, program kickoff meetings, program annual meetings, and annual innovation summits;
• Lists of potential collaborations that have derived from, and can be traced to, ARPA-E meetings;
• Proposals (funded/nonfunded) that show evidence of collaboration; and
• Lists of technologies and/or commercialized outcomes, revenues, proposals funded by ARPA-E or other entities, prizes awarded, and submissions by collaborators or communities resulting from or linked to ARPA-E organized meetings.

To facilitate contact with necessary individuals and organizations, the Committee respectfully requests:

• List of ARPA-E program directors for consultations or to participate in focus groups;
• Résumés of past and present program directors and deputy directors, as well as personnel within ARPA-E; and
• List of ARPA-E stakeholders to interview including other federal programs or industries that benefit from ARPA-E funded research.

I would appreciate your best efforts to provide the requested information by October 15, 2015. The Committee understands that several of the items above may require further discussion and that some will require careful attention and proper handling to protect confidential information. As I mentioned earlier, the Committee and I stand ready to work with ARPA-E to ensure that protection and note that the National Academies has policies and procedures in place to work with such information, providing the protection that ARPA-E and its applicants require.

I look forward to your response.

Sincerely,

Pradeep Khosla, Chair

Committee for the Evaluation of ARPA-E