Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force

Consensus

Download PDF

Paperback - $37.80
Ebook - $30.59

Request an exam or desk copy

The Department of the Air Force (DAF) is in the early stages of incorporating modern artificial intelligence (AI) technologies into its systems and operations. The integration of AI-enabled capabilities across the DAF will accelerate over the next few years.

At the request of DAF Air and Space Forces, this report examines the Air Force Test Center technical capabilities and capacity to conduct rigorous and objective tests, evaluations, and assessments of AI-enabled systems under operational conditions and against realistic threats. This report explores both the opportunities and challenges inherent in integrating AI at speed and at scale across the DAF.

Additional materials

Public Briefing Slides

Publication information

194 pages · 7 x 10 · paperback
ISBN Paperback: 0-309-70439-1
ISBN Ebook: 0-309-70440-5
DOI: https://doi.org/10.17226/27092

Suggested citation

National Academies of Sciences, Engineering, and Medicine. 2023. Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force. Washington, DC: The National Academies Press.

Import this citation to:

May Casterline

May Casterline is an image scientist and software developer with a background in satellite and airborne imaging systems. Her research interests include deep learning, hyperspectral and multispectral imaging, innovative applications of machine learning approaches to remote sensing data, multimodal data fusion, data workflow design, high performance computing applications, and creative software solutions to challenging geospatial problems. She holds a PhD and Bachelors of Science in Imaging Science from Rochester Institute of Technology, with a focus on remote sensing. In industry she has acted as a product owner, technical lead, lead developer, and image scientist on both research initiatives and development projects. Dr. Casterline joined the NVIDIA federal solution architecture team in 2017 focusing on deep learning and artificial intelligence applications.

Thomas A. Longstaff

Thomas A. Longstaff is the Chief Technology Officer of the SEI. As CTO, Longstaff is responsible for formulating a technical strategy and leading the funded research program of the institute based on current and predicted future trends in technology, government, and industry. Before joining the SEI as CTO in 2018, Longstaff was a program manager and principal cybersecurity strategist for the Asymmetric Operations Sector of the Johns Hopkins University Applied Physics Laboratory (APL), where he led projects on behalf of the U.S. government, including nuclear command and control, automated incident response, technology transition of cyber R&D, information assurance, intelligence, and global information networks. He also was chair of the Computer Science, Cybersecurity, and Information Systems Engineering Programs and co-chair of Data Science in the Whiting School at Johns Hopkins. His academic publications span topics such as malware analysis, information survivability, insider threat, intruder modeling, and intrusion detection. He maintains an active role in the information assurance community and regularly advises organizations on the future of network threat and information assurance. He is an editor for Computers and Security, and has previously served as associate editor for IEEE Security and Privacy; general chair for the New Security Paradigms Workshop and Homeland Security Technology Conference; and numerous other program and advisory committees. Prior to joining the staff at APL, Longstaff was the deputy director for technology for the CERT Division at the Software Engineering Institute. In his 15-year tenure at the SEI CERT Division, he helped create many of the projects and centers that made the program an internationally recognized network security organization. His work included assisting the Department of Homeland Security and other agencies to use response and vulnerability data to define and direct a research and operations program in analysis and prediction of network security and cyber terrorism events.
Longstaff received his bachelor’s degree in physics and mathematics from Boston University and his master’s degree in applied science and his Ph.D. in computer science from the University of California, Davis.

Craig R. Baker

Craig R. Baker is the President, Baker Development Group, LLC, consulting, leadership, and teaching. He is a trusted executive leader widely known as a strategic planner and executor of large, important, highly visible projects and products to mitigate risks. Mr. Baker retired from the US Air Force as a Brigadier General in July 2021. He graduated from the US Military Academy at West Point in 1992, was a command, combat, test pilot, and instructed at the US Air Force Fighter Weapons School (“Top Gun”). Mr. Baker commanded at both the squadron and wing levels. Additionally, he was the Technical Director GM/Test Program Manager, 59th Test & Evaluation squadron achieving $500M in savings cultivated and lives saved by creating and establishing an innovative process and software program that required 60% fewer assets and personnel; and 60% of worldwide objectives met in developing and integrating a Congressionally directed fleet capability while delivering a historical first milestone highlighted to the enterprise president weekly. Mr. Baker led multi-service weapons assessment teams into Iraq and Afghanistan after OIF and OEF which resulted in revolutionary software program and new weapon developments. Lastly, he earned two Master of Science degrees in Strategic Intelligence and Strategic Studies.

Robert A. Bond

Robert A. Bond, prior to his appointment as Principal Staff, served for five years as Lincoln Laboratory's Chief Technology Officer. He was formerly the Associate Head of the Intelligence, Surveillance, Reconnaissance (ISR) and Tactical Systems Division. In his 42-year career, he has led research initiatives in very-large-scale integrated (VLSI) circuits, software technology, parallel processors, adaptive and nonlinear signal processing, artificial intelligence (AI), C2ISR systems, and big-data analytics.
Mr. Bond joined Lincoln Laboratory in 1987 and led the software and integration activities for the Radar Surveillance Technology Experimental Radar. In the1990s, he conducted seminal studies on the use of massively parallel processors (MPP) for real-time signal processing. He then pioneered the development of a custom VLSI processor and a 1000-node MPP for radar space-time adaptive processing. He led the development of a middleware technology for portable and scalable parallel signal processors that evolved into the Parallel Vector Tile Optimized Library (PVTOL), which won an R&D 100 award. In 2003, he received the Laboratory’s prestigious Technical Excellence Award, for his "technical vision and leadership in the application of high-performance embedded processing architectures to real-time digital signal processing systems." Since 2015, Mr. Bond has led the Laboratory’s strategic initiatives in AI and autonomous systems. As CTO, he oversaw and funded the applied research portfolios in these areas. In 2018 he founded the Recent Advances in AI for National Security (RAAINS) workshop. He is currently the Laboratory’s program manager for the Air Force - MIT AI Accelerator program. Mr. Bond has a B.S. (honors) in Physics, and is a member of the AAAI and a senior member of the IEEE.

Ramalingam Chellappa

Rama Chellappa is a Bloomberg Distinguished Professor of electrical and computer engineering and biomedical engineering with Johns Hopkins University (JHU), Baltimore. He holds a secondary appointment in the computer science department. He received the B.E. (Hons.) degree from the University of Madras, Madras, India, in 1975, the M.E. (Distinction) degree from the Indian Institute of Science, Bangalore, India, in 1977, and the M.S.E.E. and Ph.D. degrees in electrical engineering from Purdue University in 1978 and 1981, respectively. He is also affiliated with the Institute for Assured Autonomy, Center for Imaging Science, the Center for Language and Speech Processing, The Malone Center for Engineering Health, and the Mathematical Institute for Data Sciences at JHU. Before moving to JHU in August 2020, Dr. Chellappa was a Distinguished University Professor and a Minta Martin professor of engineering with the University of Maryland (UMD). He was a Professor in the department of electrical and computer engineering and the University of Maryland Institute for Advanced Computer Studies. Currently, he holds the position of a College Park Professor in the electrical and computer engineering department at UMD. Prior to joining the University of Maryland, he was an assistant (1981–1986) and associate professor (1986–1991) and director of the Signal and Image Processing Institute (1988–1990) with the University of Southern California, Los Angeles. Over the last 41 years, he has published numerous book chapters, peer-reviewed journal, and conference papers. He has coauthored and coedited books on Markov random fields, face, and gait recognition, and collected works on image processing and analysis. He has served as a co-editor-in-chief of graphical models and image processing. His current research interests are computer vision, artificial intelligence, machine learning, image processing and pattern recognition with applications in face and gait analysis, markerless motion capture; 3-D modeling from video, image and video-based recognition and exploitation, compressive sensing, and hyper spectral processing. Professor Chellappa has received several awards, including a National Science Foundation Presidential Young Investigator Award, four IBM Faculty Development Awards, an Excellence in Teaching Award from the School of Engineering at University of Southern California-Los Angeles, and several paper awards from the International Conference on Pattern Recognition and Biometrics conferences. He received the Society, Technical Achievement and Meritorious Service Awards from the IEEE Signal Processing Society. He also received the Technical Achievement and Meritorious Service Awards from the IEEE Computer Society. At the University of Maryland, he was elected as a distinguished faculty research fellow, as a distinguished scholar-teacher, received the outstanding Faculty Research Award and the Poole and Kent Teaching Award for the Senior Faculty from the College of Engineering, an Outstanding Innovator Award from the Office of Technology Commercialization and an Outstanding GEMSTONE Mentor Award. In 2010, he was recognized as an Outstanding ECE by Purdue University. He is a fellow of the AAAI, AAAS, ACM, AIMBE, IEEE, the International Association for Pattern Recognition, the National Academy of Inventors and the Optical Society of America. He has served as an associate editor for four IEEE publications and as the editor-in-chief of the IEEE Transactions on Patter Analysis and Machine Intelligence. He served as a member of the IEEE Signal Processing Society Board of Governors and as its vice president of awards and membership. He served as a general and technical program chair for several IEEE international and national conferences and workshops. He is a golden core member of the IEEE Computer Society and served a two-year term as a distinguished lecturer of the IEEE Signal Processing Society. Recently, he completed a two-year term as president of IEEE Biometrics Council. His Google Scholar h-index is 133.

Melvin Greer

Melvin Greer is an Intel Fellow and Chief Data Scientist, Americas, Intel Corporation. He is responsible for building Intel’s data science platform through graph analytics, machine learning and cognitive computing to accelerate transformation of data into a strategic asset for Public Sector and commercial enterprises. His systems and software engineering experience has resulted in patented inventions in Cloud Computing, Synthetic Biology and IoT Bio-sensors for edge analytics. He functions as a principal investigator in advanced research studies, including Nanotechnology, Additive Manufacturing and Gamification. He significantly advances the body of knowledge in basic research and critical, highly advanced engineering and scientific disciplines. Dr. Greer has been appointed to Senior Advisor and Fellow at the FBI IT and Data Division. He is charged with acceleration of the FBI mission by supporting appropriate data collection, data analytics, discovery and visualization via advanced data science and AI techniques. Dr. Greer is one of the 2018 LinkedIn Top 10 Voices in data science and analytics. He also received the Washington Exec inaugural Pinnacle Award as the 2018 Artificial Intelligence Executive of the Year. He received the 2017 BDPA Lifetime Achievement Award and the 2012 BEYA Technologist of the Year Award which recognize his outstanding technical contributions that have had a material impact and high value to society as a whole. Dr. Greer has been appointed Fellow of the National Cybersecurity Institute where he assists government, industry, military, and academic sectors meet the challenges in cyber security policy, technology and education. Dr. Greer is Professor, Master of Science in Data Science program at Southern Methodist University (SMU) and Adjunct Faculty, Advanced Academic Program at Johns Hopkins University, where he teaches the Masters of Science course “Practical Applications of Artificial Intelligence”. In addition to his professional and investment roles, he is Founder and Managing Director of the Greer Institute for Leadership and Innovation, focused on research and deployment of a 21st Century Leadership Model. Dr. Greer is a frequent speaker at conferences and universities and is an accomplished author; his fifth book “Practical Cloud Security a Cross Industry View” is his most recently published book. Melvin is a Board of Director at National GEM Consortium where he oversees and aligns its strategic direction, educational policy, finances and operations with the mission of the Fellowship program. As a popular educator and board member at a number of Historical Black Colleges and Universities, Dr. Greer is leading science, technology, mathematical and engineering (STEM) research initiatives, directly trying to shape a more diverse generation of up-and-coming technical talent.

Tamara G. Kolda

Tamara G. Kolda is an Independent Mathematical Consultant under the auspices of her company MathSci.ai based in California. She is also a Distinguished Visiting Professor in the Department of Industrial Engineering & Management Science at Northwestern University in Evanston, Illinois. From 1999-2021, she was a researcher at Sandia National Laboratories in Livermore, California. She specializes in mathematical algorithms and computation methods for tensor decompositions, tensor eigenvalues, graph algorithms, randomized algorithms, machine learning, network science, numerical optimization, and distributed and parallel computing. She is a member of the National Academy of Engineering (NAE), Fellow of the Society for Industrial and Applied Mathematics (SIAM), and Fellow of the Association for Computing Machinery (ACM). She holds a PhD in Applied Mathematics from the University of Maryland at College Park.

Robin R. Murphy

Robin R. Murphy is the Raytheon Professor of Computer Science and Engineering at Texas A&M University and a director of the Center for Robot-Assisted Search and Rescue. Her research focuses on artificial intelligence, robotics, and human-robot interaction for emergency management. She has deployed ground, aerial, and marine robots to over 30 disasters in five countries including the 9/11 World Trade Center, Fukushima, Hurricane Harvey, and the Surfside collapse. She is an AAAS, ACM and IEEE Fellow, a TED speaker, and her contributions to robotics have been recognized with the ACM Eugene L. Lawler Award for Humanitarian Contributions and a US Air Force Exemplary Civilian Service Award medal. She holds a Ph.D. (’92) and M.S. (‘89) in computer science and B.M.E. (‘80) in mechanical engineering, all from the Georgia Institute of Technology.

David S. Rosenblum

David S. Rosenblum is the Planning Research Corporation Professor and Chair of the Department of Computer Science at George Mason University. Since receiving his Ph.D., he has held positions as Member of the Technical Staff at AT&T Bell Laboratories (Murray Hill); Associate Professor and Associate Chair at University of California, Irvine; CTO and Principal Architect at PreCache, Inc.; Professor of Software Systems at University College London; and Provost’s Chair Professor, Dean of the School of Computing, and Founding Director of the NUS-Singtel Cyber Security R&D Lab at National University of Singapore. He has made significant contributions to a broad array of research problem areas in computer science, including software engineering, distributed systems, ubiquitous computing, and machine learning. Among his most highly cited research are works on internet-scale publish/subscribe computing; assertion processing techniques and regression testing methods for software engineering; and machine learning and deep learning techniques for recommendation systems, activity recognition, and social media analytics. He is a Fellow of the ACM and IEEE and has received two 10-year test-of-time awards, including the ICSE 2002 Most Influential Paper Award for his ICSE 1992 paper on assertion processing, and the inaugural 2008 ACM SIGSOFT Impact Paper Award for his ESEC/FSE 1997 paper on internet-scale event observation and notification (co-authored with Alexander L. Wolf). He also received the 2018 ACM SIGSOFT Distinguished Service Award. He received his Ph.D. from Stanford University.

John N. Shanahan

John N.T. "Jack" Shanahan, United States Air Force, Retired, retired in 2020 after a 36-year military career. In his final assignment, he served as the inaugural Director of the U.S. Department of Defense (DoD) Joint Artificial Center (JAIC). Jack served in a variety of operational and staff positions in various fields including flying, intelligence, policy, and command and control. Jack established and led DoD’s pathfinder AI fielding program (Project Maven). Jack is an Adjunct Senior Fellow with the Technology and National Security Program at the Center for a New American Security. He is a member of the Institute of Electrical and Electronics Engineers Standards Association (IEEE-SA) Autonomous Weapons Systems Assurance and Safety Subcommittee

Rebecca Willett

Rebecca Willett is a Professor of Statistics and Computer Science at the University of Chicago. Her research is focused on machine learning, signal processing, and large-scale data science. Willett received the National Science Foundation CAREER Award in 2007, was a member of the DARPA Computer Science Study Group, received an Air Force Office of Scientific Research Young Investigator Program award in 2010, was named a Fellow of the Society of Industrial and Applied Mathematics in 2021, and was named a Fellow of the IEEE in 2022. She is a co-principal investigator and member of the Executive Committee for the Institute for the Foundations of Data Science, helps direct the Air Force Research Lab University Center of Excellence on Machine Learning, and currently leads the University of Chicago’s AI+Science Initiative. She serves on advisory committees for the National Science Foundation’s Institute for Mathematical and Statistical Innovation, the AI for Science Committee for the US Department of Energy’s Advanced Scientific Computing Research program, the Sandia National Laboratories Computing and Information Sciences Program, and the University of Tokyo Institute for AI and Beyond. She completed her PhD in Electrical and Computer Engineering at Rice University in 2005 and was an Assistant then tenured Associate Professor of Electrical and Computer Engineering at Duke University from 2005 to 2013. She was an Associate Professor of Electrical and Computer Engineering, Harvey D. Spangler Faculty Scholar, and Fellow of the Wisconsin Institutes for Discovery at the University of Wisconsin-Madison from 2013 to 2018.

Meeting

Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force Meeting #3

August 23, 2022

11:00 AM - 4:30 PM (ET)

Past

Agenda

Download the Agenda

About

This is the third committee meeting. This will be a data gathering meeting.

Additional Information

Is it a Closed Session event?

Not Closed

Contact

Ryan Murphy

rmurphy@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Log In

Sign in to access your saved publications, downloads, and email preferences.

Former MyNAP users: You'll need to reset your password on your first login to MyAcademies. Click "Forgot password" below to receive a reset link via email. Having trouble? Visit our FAQ page to contact support.

Members of the National Academy of Sciences, National Academy of Engineering, or National Academy of Medicine should log in through their respective Academy portals.

Email (required)
Password (required)

Don't have an account?

Register

Email (required)
New password (required)
Retype password (required) Password must be at least 8 characters
I accept the terms of use
terms of use

Thank You

Thank you for creating a MyAcademies account!

Enjoy free access to thousands of National Academies' publications, a 10% discount off every purchase, and build your personal library.

Forgot Password

Enter the email address for your MyAcademies (formerly MyNAP) account to receive password reset instructions.

Reset Requested

We sent password reset instructions to your email . Follow the link in that email to create a new password. Didn't receive it? Check your spam folder or contact us for assistance.

Reset Password

New password (required)
Retype password (required) Password must be at least 8 characters

Reset Password

Your password has been reset.

Log In

Verify email

Verify Your Email Address

We sent a verification link to your email. Please check your inbox (and spam folder) and follow the link to verify your email address. If you did not receive the email, you can request a new verification link below

Subscribe

First Name
Last Name
Email (required)
I agree to receive emails from the National Academies (required)

We respect your privacy and will never share your information.

Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force

Featured publication

Description

Contributors

Committee

Sponsors

Staff

Parent projects

Air Force Studies Board

Major units and sub-units

Contact

Contact us

Provide feedback

Request materials

More like this

Review of the draft United by Nature assessment

Modernizing Mathematics Education for Grades 9-14

Health Care Financing to Improve Availability of Affordable Contraception: Identifying Promising Practices

Earth Systems and Resources Program Area

Social and Economic Systems Program Area

Computing Research, Technologies, and Systems Program Area

Physical Sciences, Systems, and Infrastructure Program Area

Modernizing Mathematics Education for Grades 9-14

Improving Command Climate, Unit Cohesion, and Warfighter Readiness

Additional materials

Publication information

Suggested citation

About

Additional Information

Contact

Ryan Murphy

Disclaimer

My Academies

Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force

Dropdown items

Featured publication

Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force

Description

Contributors

Committee

Sponsors

Staff

Parent projects

Air Force Studies Board

Major units and sub-units

Contact

Contact us

Provide feedback

Request materials

More like this

Review of the draft United by Nature assessment

Modernizing Mathematics Education for Grades 9-14

Health Care Financing to Improve Availability of Affordable Contraception: Identifying Promising Practices

Earth Systems and Resources Program Area

Social and Economic Systems Program Area

Computing Research, Technologies, and Systems Program Area

Physical Sciences, Systems, and Infrastructure Program Area

Modernizing Mathematics Education for Grades 9-14

Improving Command Climate, Unit Cohesion, and Warfighter Readiness