Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force (2023)
Chapter: 7 Concluding Thoughts
7
Concluding Thoughts
The committee was tasked with the following questions:
- Evaluate and contrast current testing and assessment methods employed by the Department of the Air Force (DAF) and in commercial industry.
- Consider examples of artificial intelligence (AI) corruption under operational conditions and against malicious cyberattacks.
- Recommend promising areas of science and technology that may lead to improved detection and mitigation of AI corruption.
What was uncovered through investigating these questions was a significant overlap between the answers. For example, examples of AI corruption (question 2) were a common topic of promising areas of science and technology (question 3) and reflected as an issue in the current testing and assessment methods employed by the DAF (question 1).
In structuring this report, the committee organized chapters such that each question was primarily addressed in a specific chapter in the report, and thus the recommendations were primarily reported in those sections. However, each chapter of the report contains relevant findings and recommendations for each of the questions addressed by the committee, so in practice, it was not possible to isolate the questions to individual chapters.
Task 1, “Evaluate and contrast current testing and assessment methods employed by the Department of the Air Force and in commercial industry,” is primarily answered in Chapters 3 and 4. In these chapters, the current testing and
assessment methods found by the committee are described and referenced, and a comparison to best practices in commercial industry are directly covered. However, to fully understand this contrast, findings and recommendations from Chapters 2 and 5 must be also considered.
Task 2, “Consider examples of AI corruption under operational conditions and against malicious cyberattacks,” is primarily addressed in Chapter 5. However, the topic of AI corruption is a primary challenge throughout all of the test and evaluation (T&E) of AI-enabled systems and thus is mentioned throughout the study, especially in Chapter 6.
Task 3, “Recommend promising areas of science and technology that may lead to improved detection and mitigation of AI corruption,” is primarily addressed in Chapter 6. However, AI corruption is under active S&T research and in the spirit of DevSecOps and AIOps, solutions are being deployed as rapidly as they are discovered. Thus, many of the approaches in Chapters 3, 4, and 5 also support findings and recommendations to mitigate AI corruption.
In short, the complexity, interconnection, and coupling of issues throughout T&E with AI-enabled systems will require a reassessment of all T&E policies, processes, and procedures to assure that validation and verification of all systems, not just the AI components, will support the necessities of a dynamic and risky deployment and operational environment. In this case, the committee concluded that while the questions appeared quite straightforward on an initial reading, in fact the DAF has now caught the AI tiger by its tail. Taming that tiger will be challenging, especially as AI-enabled components become commonplace in all platforms and MDAPs, but it is not at all an insurmountable problem. It requires vision, hands-on leadership, prioritization, and a shared commitment to an AI-enabled future DAF.
