Over the past 2 decades, computing and communications networks have made it possible for decision makers to access enormous amounts of information. Entire libraries are available for searching, multiple streams of sensed data may be tapped, and collections of other network users can in many cases be accessed readily. Machine learning and natural language processing have advanced to the point where they can often provide basic pre-processing of diverse types of data. Computational support in the form of large-scale data collection and analysis, visualization, etc. has been readily incorporated into some human decision making processes. For example, computation is in the control processes for all manner of processing plants (chemical processing, nuclear power generation and petroleum refining), infrastructure (electric grid and telecommunications), manufacturing (chip fabrication and large scale baking plants), assembly (electronics and automotive robotic assembly), transactions (credit card and banking) and the military (management of theater operations).
However, the ease with which humans have already integrated computational systems into decision making ranging from ordinary to critical, from simple to complex, belies a deeper truth: this area of inquiry is still in its infancy relative to where multi-disciplinary research could take it over the next generation. This state of affairs has generated an environment that is ripe for a rethinking of human-computer collaboration in the context of complex decision making. The vast amount of information that can be brought to bear does not guarantee better decisions or a more straightforward or reliable decision-making process. How to take advantage of these capabilities is the subject of this report of the Committee on Integrating Humans, Machines and Networks: A Global Review of Data-to-Decision Technologies.
The multidisciplinary committee that was formed at the request of the National Ground Intelligence Center of the U.S. Army (NGIC) included experts in autonomous agents, cognitive science, decision analysis, machine learning, neuroscience, statistics, and other areas. The sponsor wanted to better understand how enabling technologies are being integrated to inform and improve computer-assisted decision making; what some of the impediments are to their integration; and obtain a sense of the research that is occurring in university, government, and industrial labs inside and outside of the United States.
Early on in its deliberations, the committee perceived the unbounded nature of this broad topic—that is, the more they learned as a group about the varied aspects of human-machine collaboration for decision making, the more there was to study. Thus the committee concluded that a useful contribution to this topic at this time would be a preliminary exploration of the issues that could provide a roadmap for future multidisciplinary research. The report’s structure takes a linear path: from human decision making; to relevant new computing capabilities; to emerging explorations of human-computer team decision making; to several research challenges that need to be overcome in order to realize the next steps in human-machine collaboration for complex decision making.
Following are the committee’s findings. They are listed in the order that they appear in the text:
Finding 1: A common representation of the decision-making process, used to train fighter pilots in rapid decision making for air combat, calls for sequential steps to observe, update beliefs, choose an action, and take the action (the so-called OODA loop). While those steps are inherent to any careful decision making, for complex decisions the OODA loop framework does not readily reflect feedback loops between the steps and branching to consider multiple le choices of action, both of which are common. The study of decision making in complex situations, and the design of automated decision support systems, requires an understanding of those complexities. Thus the OODA-loop framework may not be sufficient in those contexts.
Finding 2: Increasingly the data used to support computer-assisted decisions are drawn from heterogeneous sources (e.g. unstructured text, images, simulation outputs). Current techniques for filtering and aggregating these disparate data types into a well-characterized input for decision making are limited, which therefore limits the quality of the decisions.
Finding 3: While improved information availability can improve the quality of decision making, more information alone is not sufficient. This is particularly evident in complex scenarios where the goals of different team members are not completely aligned and delays make it difficult to attribute effects to actions.
Finding 4: Computer assists to human decision making will “come of age” when some of the computational elements are not simply assistive, but perform at a level that they are trusted as “near-peer” teammates in an integrated human-computer system. One of the key challenges of this integration will be the development of new techniques for test and evaluation that build trust between the human partner and the computational elements.
Finding 5: Humans and computation have different strengths in what they accomplish and there are several aspects of human decision making that can benefit from computer-aided systems, such as cognition, recognition of errors in judgment and task allocation. Similarly, there are several aspects of computer processing that can benefit from human guidance, such as prioritization, dealing with unusual or unexpected situations, understanding social and cultural context, and taking environmental and contextual information into account. The committee finds that the computational assists to human decision making are best when the human is thought of as a partner in solving problems and executing decision processes, where the strengths and benefits of machine and humans are treated as complementary co-systems.
In addition to these findings, the committee identified a number of promising research directions to improve the scientific basis for strong human-computer decision making:
- Data-to-decisions is an umbrella term that is not clearly defined. We need a better understanding of how cognitive functions can be supported over time and in context and an overall framework for thinking about how to design human-computer decision systems;
- The ubiquitous capability to capture, store, reproduce, move, and reuse data has led to decisions increasingly being made by networks composed of humans and machines. Yet, the exploitation of that data is often ad hoc. Research is needed to frame and systematize how we exploit that data;
- At any moment, whether a particular datum will be relevant or irrelevant into the future is task- and context-dependent, so there is an incentive to retain more, rather than less. Thus, a key challenge is to build task and context models that enable data to be filtered and processed into “useful information”;
- Another challenge is developing systems that allow both humans and computers to work together in a harmonious team, rather than one supervising the other. This requires research to help individual and team exploration of (partial and incomplete) hypotheses, to enable continuous learning by the system (e.g., so the system can learn how to predict an analyst’s needs and preferences); to guide continuous ingesting of data and its metadata and fusing it into the existing data; to cue decision makers to relevant, unexplored data or behavior; and to facilitate the sharing of hypotheses and derived knowledge among team members (such as by developing languages that make it easy for decision makers to state what they want the data to tell them). Creating harmonious human-computer teams would also be helped by research in comparing the different roles of humans and computers in mixed teams;
- Complex decision making often takes place in a complex environment, with multiple activities occurring simultaneously. This leads to frequent interruptions and the need to switch tasks and revise priorities. Current human-computer systems do not handle interruptions well and they need to provide more support for the resumption of interrupted activities. More research is needed on computational-interruption management techniques and algorithms, rooted in an understanding of people’s cognitive and attentional capabilities; and
- More work is needed to develop a methodology for evaluating and assigning metrics for each individual piece of the collaboration and for the quality of the decisions made by the overall human-machine collaborative system.
The committee anticipates that human-computer decision-making systems will continue to advance, but this outcome is not certain. A concerted and thoughtfully guided effort will improve the chances of success.
