1
Introduction

This introductory chapter describes the assessment process conducted by the National Academies of Sciences, Engineering, and Medicine’s Army Research Laboratory Technical Assessment Board (ARLTAB). It then describes the preparation and organization of the report, the assessment criteria, and the approach taken during the report preparation.

ARLTAB is guided by the following statement of task:

The charge of ARLTAB is to provide assessments of the scientific and technical quality of ARL. These assessments include the development of findings and recommendations related to the quality of ARL’s research, development, and analysis initiatives. For this 2022 assessment, ARLTAB was charged to review the work in ARL’s three competencies: humans in complex systems, terminal effects, and weapons sciences.

To conduct its assessments, ARLTAB was assisted by National Academies’ panels, each of which focused on one of ARL’s competencies. ARLTAB’s assessments are commissioned by ARL itself rather than by one of its parent organizations. For this assessment, ARLTAB consisted of eight leading scientists and engineers whose collective experience spans the main topics within ARL’s 2022 scope. Three panels, each of which focuses on one or more of ARL’s research competencies, report to ARLTAB. Three of the ARLTAB members serve as chairs of these panels. The panels range in size from 9 to 10 members, whose expertise is carefully matched to the technical fields covered by the areas that they review. In total, 32 members participated in the reviews that led to this report. All panel and ARLTAB members participate without compensation.

The National Academies appointed the ARLTAB and panel members with an eye to assembling a slate of experts without conflicts of interest and with balanced perspectives. The experts include current and former executives and research staff from industrial research and development (R&D) laboratories, leading academic researchers, and staff from national laboratories and federally funded R&D centers. Seventeen are members of the National Academy of Engineering and four are members of the National Academy of Sciences. A number have been leaders in relevant professional societies. ARLTAB and its panels are supported by National Academies staff, who interact with ARL on a continuing basis to ensure that ARLTAB and the panels receive the information they need to carry out their assessments. The

ARLTAB and panel members typically serve for finite terms, generally 1.5 to 6 years, so that viewpoints are regularly refreshed and the expertise of the ARLTAB and panel members continues to match ARL’s activities. Biographical information on the ARLTAB members appears in Appendix A.

PREPARATION OF THIS REPORT

The National Academies have been producing assessments of ARL since 1996. As with the earlier reviews, this report contains ARLTAB’s perceptions about the quality of ARL’s work. (Chapters 2 through 4 focus on the humans in complex systems, terminal effects, and weapons sciences competencies areas and Chapter 5 provides conclusions and recommendations for these three competencies, along with conclusions and recommendations that address crosscutting themes identified within these three competencies.) The rest of this chapter explains the rich set of interactions that supports those judgments.

This review is based on a large amount of information received from ARL and on interactions between ARL staff and ARLTAB and its panels. The amount of information that is funneled to ARLTAB, including the evaluations by the recognized experts who make up ARLTAB’s panels, provides a solid foundation for a thorough peer review. Most of the information exchange occurs during the annual meetings convened by the respective panels at the appropriate ARL sites, both in formal interactions during technical presentations and in less formal interactions during poster sessions and during joint lunch or dinner sessions.

For this 2022 review, ARL displayed a very healthy level of information exchange and acceptance of external comments. The assessment panels and ARLTAB engaged in many constructive interactions with ARL staff during their virtual meeting and site visits in 2022. Each panel’s review meeting lasted between 2.5 and 3.5 days, during which time the panel members received a combination of overview briefings by ARL management and technical briefings by ARL staff. Prior to the meetings, the panels received extensive materials for review. Useful collegial exchanges took place between panel members and individual ARL investigators during the site visits as ARL staff members sought clarification about panel comments or questions and drew on panel members’ contacts and sources of information.

The overview briefings brought the panels up to date on the broad scope of ARL’s scientific and technical work and its recent reorganization in October 2022. This context-building step was needed because the panels are purposely composed of people who, while experts in the technical fields covered by ARL’s competencies that they reviewed, were not engaged in collaborative work with ARL. Technical briefings for the panels focused on R&D goals, strategies, methodologies, and results of selected projects at the laboratory. Briefings were targeted at coverage of a representative sample of each of ARL’s research core competencies within their respective competencies. Briefings included poster sessions that allowed direct interaction among the panelists and staff of projects that were not covered in the briefings.¹ Classified sessions were held as necessary, and the panels also received Controlled Unclassified Information.

Ample time during both the overview and the technical briefings was devoted to discussion, which enabled panel members to pose questions and ARL staff to provide additional technical and contextual information to clarify panel members’ understanding. The panels also devoted sufficient time to closed session deliberations, during which they developed findings and identified important questions or gaps in panel understanding. Those questions or gaps were discussed during follow-up sessions with ARL staff so that each panel was confident of the accuracy and completeness of its assessments. Panel members continued to refine their findings, conclusions, and recommendations during written exchanges among themselves after the meetings.

In addition to the insights that they gained from the panel meetings, ARLTAB members received exposure to ARL and its staff at the ARLTAB fall virtual meeting. The 2022 ARLTAB meetings refined

___________________

¹ Agendas of the public portions of the panel meetings can be found on the National Academies’ website at https://www.nationalacademies.org/lab/laboratory-assessments-board.

elements of the assessment process focused on ARL’s competencies, including a discussion with ARL on the read-ahead materials, review agendas, and expertise required within the panels.

ASSESSMENT CRITERIA

During the assessment, ARLTAB and its panels considered the following requests posed by the ARL director. These requests form the basis of the entire assessment:

1. Please comment on the scientific quality of the research within the competency, particularly in comparison to that of other leading research institutions and funding agencies, both nationally and internationally.
2. Please comment on how the competency demonstrates and reflects a broad understanding of the underlying science and research conducted elsewhere.
3. Please comment on the qualifications of the teams of ARL and extramural scientists and engineers supporting the competency.
4. Please comment on the quality, appropriateness, and utilization of facilities, equipment, information technology, and digital infrastructure.
5. Please comment on the soundness of the competency’s research methods and methodologies (e.g., the reasoning; the use of theory, modeling and simulation, analysis, and experimentation).
6. Please comment on opportunities, if found, within the competency where improved direction, increased focus, connections with other research lines, or other changes could better and more quickly address the competency objectives.
7. Please comment on any specific areas, if found, where the research may be at major risk of not meeting its objectives and provide reasoning.
8. Please comment on the overall balance of the competency’s research portfolio (e.g., core competencies, partnerships, supporting extramural partners, etc.) to address the cumulative competency goals.
9. Please comment on any promising new areas of research or novel research approaches the competency should pursue.

Specifically excluded from the assessment criteria are the relevance to, and impact of, the scientific and technical work with respect to Army mission.

The answers to these requests were reorganized under the following section headers for Chapters 2 through 4:

• Technical Quality of the Core Competency Research (answers requests a, b, e, f, g, h, and i)
• Competency Portfolio of Scientific Expertise (answers request c)
• Competency Facilities and Resources (answers request d)
• Overall Scientific Quality of the Competency (an overarching summary at the competency level containing elements of the assessment criteria)
• Conclusions and Recommendations (contains the most salient and actionable suggestions from the chapter)

APPROACH TAKEN DURING REPORT PREPARATION

This report represents ARLTAB’s consensus findings and recommendations, developed through deliberations, which included consideration of the notes prepared by the panel members summarizing their assessments. As mentioned, the chair of each panel is a member of ARLTAB, and in that capacity

they bring their detailed knowledge of their panel’s site visit and deeper discussions. ARLTAB’s aim with this report is to provide guidance based on the assessment criteria to the ARL director that will help ARL sustain its process of continuous improvement, as well as guidance to those managing the competencies. To that end, ARLTAB examined its extensive and detailed notes from the many ARLTAB panel and individual interactions with ARL during 2022. From those notes, it distilled a shorter list of the main trends, opportunities, and challenges that merit attention at the level of the ARL director and the management team. ARLTAB used that list as the basis for this report. Specific ARL projects are used to illustrate these points in the following chapters when it is helpful to do so, but ARLTAB did not aim to present the director with a detailed account of interactions with bench scientists. The draft of this report was subsequently honed and reviewed according to the National Academies’ procedures before being released.

ARLTAB applied a largely qualitative rather than quantitative approach to the assessment. The approach of ARLTAB and its panels relied on the experience, technical knowledge, and expertise of its members, whose backgrounds were carefully matched to the three competency areas that were the focus of the 2022 review. ARLTAB and its panels reviewed selected examples of the scientific and technological research performed by ARL; it was not possible to review all ARL programs and projects exhaustively. Given the necessarily non-exhaustive nature of the review process, the omission of mention of any particular program or project should not be interpreted as a negative reflection on the omitted program or project.

In line with the assessment criteria, ARLTAB’s goal was to identify and report salient examples of accomplishments and opportunities for further improvement with respect to the technical merit of ARL work and specific elements of ARL’s resource infrastructure that are intended to support the technical work. Both the Army Research Office’s (ARO’s) and the Army Research Directorate’s (ARD’s) extramural and intramural work exist, and often integrate, together under the holistic umbrella of each competency and its core competencies. Under the advisement of ARL, the approach to this assessment was not to tease out what was ARD versus ARO’s research, but rather to review the competency and its core competencies without specific focus on whether the research originated from ARO or ARD. In all cases when the work is described as “extramural” in the upcoming chapters, it is important to note that such work is still considered by ARL to be “intramural” because ARL is overseeing this work with their extramural partners.

Collectively, these highlighted examples for each ARL research competency and its core competencies are intended to portray an overall impression of the laboratory while preserving useful mention of suggestions specific to projects and programs that ARLTAB considered to be of special note within the set of those examined.

REPORT CONTENT

This chapter has addressed assessment process used by ARLTAB and its three panels. Chapters 2 through 4 provide detailed assessments of the competencies reviewed during 2022. Chapter 5 presents conclusions and recommendations that cut across the three competencies. The appendixes provide ARL’s biographical information on the ARLTAB members, research core competencies, the assessment criteria used by ARLTAB and its panels, and a list of acronyms and abbreviations found in this report.