C


Assessment Criteria

The Army Research Laboratory Technical Assessment Board’s (ARLTAB’s) assessment considered the following general questions posed by the Army Research Laboratory (ARL) director:

• Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally?
• Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?
• Does the research employ the appropriate laboratory equipment and numerical models?
• Are the qualifications of the research team compatible with the research challenge?
• Are the facilities and laboratory equipment state-of-the-art?
• Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?

To assist ARL in addressing promising technical approaches, the ARLTAB also considered the following questions:

• Are there especially promising projects that, with improved direction or resources, could produce outstanding results that can be transitioned ultimately to the field?
• Are there promising outside-the-box concepts that should be pursued but are not currently in the ARL portfolio?

The ARLTAB applied the following metrics or criteria to the assessment of the scientific and technical work reviewed at the ARL:

1. Effectiveness of interaction with the scientific and technical community

1. Papers in quality refereed journals and conference proceedings (and their citation index)
2. Presentations and colloquia
3. Participation in professional activities (society officers, conference committees, journal editors)
4. Educational outreach (serving on graduate committees, teaching/lecturing, invited talks, mentoring students)
5. Fellowships and awards (external and internal)
6. Review panel participation (ARO, NSF, MURI, and the like)
7. Recruiting new talent into ARL
8. Patents and intellectual property (and examples of how the patent or intellectual property is used)
9. Involvement in building an ARL-wide cross-directorate community
10. Public recognition, e.g., in the press and elsewhere, for ARL research

2. Formulation of projects’ goals and plans

1. Are tasks well defined to achieve objectives?
2. Does the project plan clearly identify dependencies (i.e., successes depend on success of other activities within the project or outside developments)?
3. If the project is part of a wider activity, is the role of the investigators clear and are the project tasks and objectives clearly linked to those of other related projects?
4. Are milestones identified if they are appropriate? Do they appear feasible?
5. Are obstacles and challenges defined (technical, resources)?
6. Does the project represent an area where application of ARL strengths is appropriate?

3. Research and development methodology

1. Are the hypotheses appropriately framed within the literature and theoretical context?
2. Is there a clearly identified and appropriate process for performing required analyses, prototypes, models, simulations, tests, and so on?
3. Are the methods (e.g., laboratory experiment, modeling/simulation, field test, analysis) appropriate to the problems? Do these methods integrate?
4. Is the choice of equipment/apparatus appropriate?
5. Is the data collection and analysis methodology appropriate?
6. Are conclusions supported by the results?
7. Are proposed ideas for further study reasonable?
8. Do the trade-offs between risk and potential gain appear reasonable?
9. If the project demands technological or technical innovation, are those needs being met?
10. What stopping rules, if any, are being or should be applied?

4. Capabilities and resources

1. Are the qualifications and number of the staff (scientific, technical, and administrative) appropriate to achieve success of the project?
2. Is the state of the equipment and facilities adequate?
3. If staff or equipment are not adequate, how might the project be subjected to triage (which technical thrust should be emphasized, which sacrificed?) to best move toward its stated objectives?