Strengthening Data
Science Methods
FOR DEPARTMENT OF DEFENSE
PERSONNEL AND
READINESS MISSIONS

Committee on Strengthening Data Science Methods for
Department of Defense Personnel and Readiness Missions

Committee on Applied and Theoretical Statistics

Board on Mathematical Sciences and Their Applications

Division on Engineering and Physical Sciences

A Report of

THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This project was supported by Contract No. HQ0034-10-D-0003 TO#12 with the U.S. Department of Defense and Grant No. H98230-14-1-0116 from the National Security Agency. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the view of any organization or agency that provided support for the project.

International Standard Book Number 13: 978-0-309-45078-2
International Standard Book Number 10: 0-309-45078-0
Digital Object Identifier: 10.17226/23670

Additional copies of this report are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2017 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2017. Strengthening Data Science Methods for Department of Defense Personnel and Readiness Missions. Washington, DC: The National Academies Press. doi:10.17226/23670.

COMMITTEE ON STRENGTHENING DATA SCIENCE METHODS FOR DEPARTMENT OF DEFENSE PERSONNEL AND READINESS MISSIONS

STEPHEN M. ROBINSON, NAE,¹ University of Wisconsin, Madison, Chair

PASCALE CARAYON, University of Wisconsin, Madison

DAVID CHU, Institute for Defense Analyses, Alexandria, Virginia

CYNTHIA DWORK, NAS²/NAE, Microsoft Research, Mountain View, California

TERRY P. HARRISON, Pennsylvania State University, University Park

ALAN F. KARR, RTI International, Research Triangle Park, North Carolina

SALLIE KELLER, Social Decision Analytics Laboratory, Biocomplexity Institute of Virginia Tech, Arlington

ALAIR MACLEAN, Washington State University, Vancouver

DAVID MAIER, Portland State University, Oregon

STEPHEN M. POLLOCK, NAE, University of Michigan, Ann Arbor (until June 2015)

PAUL R. SACKETT, University of Minnesota, Minneapolis

MARK S. SQUILLANTE, IBM Research, Yorktown Heights, New York

WILLIAM J. STRICKLAND, HumRRO, Alexandria, Virginia

STEVEN TADELIS, University of California, Berkeley

Staff

MICHELLE K. SCHWALBE, Senior Program Officer, Study Director

SCOTT WEIDMAN, Board Director

CHERIE CHAUVIN, Senior Program Officer

LINDA CASOLA, Senior Program Assistant

RODNEY N. HOWARD, Administrative Assistant

BETH DOLAN, Financial Manager

JONATHAN YANGER, Research Associate

CLAIRE JI, Christine Mirzayan Science and Technology Policy Graduate Fellow

___________________

¹ NAE, National Academy of Engineering.

² NAS, National Academy of Sciences.

COMMITTEE ON APPLIED AND THEORETICAL STATISTICS

CONSTANTINE GATSONIS, Brown University, Providence, Rhode Island, Chair

DEEPAK AGARWAL, LinkedIn, Mountain View, California

KATHERINE BENNETT ENSOR, Rice University, Houston, Texas

MONTSERRAT (MONTSE) FUENTES, North Carolina State University, Raleigh

ALFRED O. HERO III, University of Michigan, Ann Arbor

DAVID M. HIGDON, Social Decision Analytics Laboratory, Biocomplexity Institute of Virginia Tech, Arlington

ROBERT E. KASS, Carnegie Mellon University, Pittsburgh, Pennsylvania

JOHN LAFFERTY, University of Chicago, Illinois

XIHONG LIN, Harvard University, Cambridge, Massachusetts

JOSÉ M.F. MOURA, NAE, Carnegie Mellon University, Pittsburgh, Pennsylvania

SHARON-LISE T. NORMAND, Harvard University, Cambridge, Massachusetts

GIOVANNI PARMIGIANI, Harvard University, Cambridge, Massachusetts

ADRIAN RAFTERY, NAS, University of Washington, Seattle

LANCE WALLER, Emory University, Atlanta, Georgia

EUGENE WONG, NAE, University of California, Berkeley

Staff

MICHELLE K. SCHWALBE, Senior Program Officer

RODNEY N. HOWARD, Administrative Assistant

LINDA CASOLA, Senior Program Assistant

BETH DOLAN, Financial Manager

BOARD ON MATHEMATICAL SCIENCES AND THEIR APPLICATIONS

DONALD G. SAARI, NAS, University of California, Irvine, Chair

DOUGLAS N. ARNOLD, University of Minnesota, Minneapolis

JOHN B. BELL, NAS, Lawrence Berkeley National Laboratory, Berkeley, California

VICKI M. BIER, University of Wisconsin, Madison

JOHN R. BIRGE, NAE, University of Chicago, Illinois

RONALD R. COIFMAN, NAS, Yale University, New Haven, Connecticut

L. ANTHONY COX, JR., NAE, Cox Associates, Denver, Colorado

MARK L. GREEN, University of California, Los Angeles

PATRICIA A. JACOBS, Naval Post Graduate School, Monterey, California

BRYNA KRA, Northwestern University, Evanston, Illinois

JOSEPH A. LANGSAM, Morgan Stanley (retired), Palm Beach Gardens, Florida

SIMON A. LEVIN, NAS, Princeton University, New Jersey

ANDREW W. LO, Massachusetts Institute of Technology, Cambridge

DAVID MAIER, Portland State University, Oregon

WILLIAM A. MASSEY, Princeton University, New Jersey

JUAN C. MEZA, University of California, Merced

FRED S. ROBERTS, Rutgers University, Piscataway, New Jersey

GUILLERMO SAPIRO, Duke University, Durham, North Carolina

CARL P. SIMON, University of Michigan, Ann Arbor

KATEPALLI SREENIVASAN, NAS/NAE, New York University

ELIZABETH A. THOMPSON, NAS, University of Washington, Seattle

Staff

SCOTT WEIDMAN, Board Director

NEAL GLASSMAN, Senior Program Officer

MICHELLE K. SCHWALBE, Senior Program Officer

RODNEY N. HOWARD, Administrative Assistant

BETH DOLAN, Financial Manager

This page intentionally left blank.

Acknowledgments

This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:

Brenda Dietrich, NAE,¹ IBM,

Alexis Fink, Intel,

Christine Fox, Johns Hopkins University Applied Physics Laboratory,

Ronald D. Fricker, Jr., Virginia Polytechnic and State University,

Henry J. Hatch, NAE, U.S. Army (retired),

Carl Hoffmann, Human Capital Management and Performance LLC,

Ronald C. Kessler, NAS²/NAM,³ Harvard Medical School,

Philip Neches, NAE, Teradata,

Stephen M. Pollock, NAE, University of Michigan, and

Michael G. Rumsey, U.S. Army Research Institute (retired).

___________________

¹ National Academy of Engineering.

² National Academy of Sciences.

³ National Academy of Medicine.

Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the views presented in the report, nor did they see the final draft of the report before its release. The review of this report was overseen by Robert F. Sproull, University of Massachusetts, Amherst, NAE, who was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the committee and the institution.

The committee would like to thank the following individuals for providing input to this study:

Joseph Adams, Institute for Defense Analyses,

Amy Alrich, Institute for Defense Analyses,

Jeff Angers, Office of the Assistant Secretary of the Army, Manpower and Reserve Affairs,

Beth Asch, RAND Corporation,

Stephanie Barna, Office of the Under Secretary of Defense (Personnel & Readiness),

Mark Breckenridge, Defense Manpower Data Center,

Iveta Brigis, Google People Operations,

Kelly Buck, Defense Manpower Data Center,

Roll Buller, RAND Corporation,

Samuel Buttery, Naval Postgraduate School,

John Carbone, Office of the Assistant Secretary of the Army, Manpower and Reserve Affairs,

Linda C. Cavalluzzo, Center for Naval Analyses,

Ray Conley, RAND Corporation,

Elia Crisman, Office of the Chief of Naval Operations,

Matthew Davis, Defense Manpower Data Center,

Bruce Doll, Defense Health Agency,

Michael Dominguez, Institute for Defense Analyses,

Colin Doyle, Institute for Defense Analyses,

Mark Engelbaum, U.S. Air Force, Manpower, Personnel and Services,

Alexis Fink, Intel,

Colum Foley, IBM Smarter Workforce,

Christine Fox, Johns Hopkins Applied Physics Laboratory,

Peggy Golfin, Center for Naval Analyses,

Jay Goodwin, U.S. Army Research Institute,

Kristy Gore, RAND Corporation,

David Gregory, Center for Naval Analyses,

Jane Handley, RAND Corporation,

Kim Hepner, RAND Corporation,

Carl Hoffmann, Human Capital Management and Performance, LLC,

Stanley Horowitz, Institute for Defense Analyses,

Mike Housman, Cornerstone OnDemand,

David Hudak, Office of the Secretary of Defense,

Magdi Kamel, Naval Postgraduate School,

David Knapp, RAND Corporation,

Jennifer Kurkoski, Google People Operations,

Lance Landower, RAND Corporation,

Marco Laumanns, IBM Research,

Paul B. Lester, U.S. Army, Research Facilitation Laboratory,

Scott Lutterloh, Department of the Navy, Manpower and Reserve Affairs,

Arun Maiya, Institute for Defense Analyses,

Philip Major, Institute for Defense Analyses,

Bill Mason, Department of Defense, Office of Information,

Sean McDonald, Office of the Chief of Naval Operations,

Laura Miller, RAND Corporation,

Dave Nap, RAND Corporation,

David Nicholls, Institute for Defense Analyses,

Mary Kate Norton, Google People Operations,

Laura Odell, Institute for Defense Analyses,

Peter Olsen, U.S. Air Force, Manpower, Personnel and Services,

Julie Pechacek, Institute for Defense Analyses,

Jeff Petersen, Center for Naval Analyses,

Christopher Rice, Intel,

Susan Rose, Institute for Defense Analyses,

Mark Rosen, Center for Naval Analyses,

Johannes Royset, Naval Postgraduate School,

Jackie Ryan, IBM Smarter Workforce,

Richard Schaeffer, Office of the Assistant Secretary of Defense,

Ed Schmitz, Center for Naval Analyses,

Brandon Shapiro, Institute for Defense Analyses,

Yu-Chu Shen, Naval Postgraduate School,

Max Simkoff, Cornerstone OnDemand,

Don Simrod, Center for Naval Analyses,

Cara Sims, RAND Corporation,

Allison Taylor, Institute for Defense Analyses,

Fred Thompson, Office of the Under Secretary of Defense (Personnel & Readiness),

George Tolis, Institute for Defense Analyses,

Mike Walker, United States Military Academy at West Point,

Kayla Williams, RAND Corporation, and

John D. Winkler, RAND Corporation.

This page intentionally left blank.

Contents

SUMMARY

1 INTRODUCTION

Overview

Study Origin and Approach

Organization of the Report

References

2 OVERVIEW OF THE OFFICE OF THE UNDER SECRETARY OF DEFENSE (PERSONNEL & READINESS)

Introduction

P&R Structure and Organization

Readiness and Force Management

Health Affairs

Defense Human Resources Activity

P&R Decisions

Force of the Future

References

3 PERSONNEL AND READINESS DATA AND THEIR USE

Introduction

Current Data and Analysis Capabilities

P&R Decision Making

References

4 OVERVIEW OF DATA SCIENCE METHODS

Introduction

Data Preparation

Data Analytics

References

5 PRIVACY AND CONFIDENTIALITY

Introduction

Limitations of Privacy-Preserving Approaches

Differential Privacy

Institutional Review Boards

The Federal Statistical System’s Legal and Governance Policy

References

6 COMMERCIAL STATE OF THE ART IN HUMAN RESOURCES ANALYTICS

Introduction

Products

Practice

References

7 IDENTIFYING P&R OPPORTUNITIES AND IMPLEMENTING SOLUTIONS

Introduction

Improve Data Quality and Sharing

Enhance Data Science Methods

Strengthen P&R Workforce Capabilities in Data Science

Conclusions

References

APPENDIXES

A Acronyms

B Biographies of the Committee

C Meetings and Presentations

D Stochastic Models of Uncertainty and Mathematical Optimization Under Uncertainty