Chapter 2: Research Approach

Project Kick-Off

The team held a kick-off meeting on June 23, 2023, with the NCHRP panel. Multiple panelists expressed the timeliness of the project and its importance for helping transportation organizations make higher quality decisions. Below are the key takeaways from the conversation and their implications for the project moving forward, which guided the direction of the research from this starting point.

• Research should be “broadly applicable” with the audience being any major decision-maker at an agency.
• Non-DOT transportation agencies (transit, MPOs) should be included in the research.
• The NCHRP panel confirmed their interest in looking to other industries and internationally that may have applicable lessons for transportation leaders.
• Be open to case studies and input from those outside the executive suite.
  • The project team approach will be to start with executives but also put out feelers for interesting decision situations throughout agencies for which the primary actor might be someone further down the organization.
• The project team will insert one more informal check-in after the case studies have started but before they are complete so that the NCHRP panel can weigh in on whether they are seeing the right stuff or if they want us to cast out for some more diversity of examples.
• Ideally, case studies will not be anonymous, but the NCHRP panel understands the sensitivities around negative examples and the project team can use our judgment and the check-in to finalize this decision.

Task 1: Literature Review

Coming out of the project’s kick-off meeting with the NCRHP panel, there were several core questions the team planned to address in the literature review:

1. How do we define a “good” decision?
2. What kinds of decisions do transportation agency leaders frequently face?
3. What commonly prevents people from reaching “good” decisions?
4. What resources are available to help people make better decisions?
5. Are there other arenas that have developed ways to make better decisions that transportation leaders can learn from?

Answering each of the above questions involved a combination of desk research and interviews with current transportation agency leaders.

Literature Review Methodology

The High Street team reviewed the literature on decision theory and practice that is oriented toward characterizing best practices and lessons learned from across the public, private, and

academic sectors. The results provide a baseline of credible information about practices for making effective decisions based on the best available information from industry journals, magazines and websites, academic journals, and business support resources. Input from past and current DOT leaders supplemented the formal literature review with real-world insights from experience where appropriate.

The research on these questions was structured in terms of three project focus areas:

• Decision Theory is based on academic literature.
• Technical Decision-Support information came from summaries of tools applied in the transportation industry and other decision-making environments.
• Real-World Application relied on input and insights from current and former DOT leaders, as well as transportation-specific literature.

These three topics formed the basis of the literature review, with different sources providing the main content for each. Desk research on the academic literature provided a foundation of the concepts that would be most helpful to help agency leaders understand how we make decisions and the dynamics at work; documents and reports on current practices in the transportation industry highlighted which tools might be helpful to guide decision-makers through decision situations; and a series of focus group discussions with transportation agency leaders to inform how they defined “good” decisions, the kinds of decisions they often faced, and what outputs from the research would be most helpful. The literature review attempted to define what constituted an “ideal” decision, what disrupts ideal decisions, and then catalog the tools and approaches that can help bridge that gap.

Desk Research

Desk research to find the cited sources included the following avenues:

• “Reason, Passion, Cognition” Coursework – The preliminary concepts in this review were informed by the subject matter expertise of Linda Moya, who teaches the foundations of cognitive decision-making in the Carnegie Mellon University course, “Reason, Passion, Cognition.” The topics in this course cover the fundamentals of how people make decisions from a cognitive perspective, the sources for which informed this document’s examination of the same topic. Moya shared the peer-reviewed papers that her course is based on with the team, summarized key points, and participated in team meetings to discuss how to best integrate these concepts into the current research.
• Additional Academic Journals – Once the team determined the concepts that would be most helpful to the research and intended audience, the team sought out additional peer-reviewed journal articles to provide a full and robust view of key concepts.
• Transport Research International Documentation – To supplement general academic perspectives on decision-making, the team explored existing research on decision-making in the transportation sector by searching the Transport Research International Documentation (TRID) database.
• Internet Research on Applied Decision-Making – Additional web research supplemented the foundation of academic peer-reviewed papers and transportation-specific resources, particularly to incorporate the wide body of work about decision-making in the realms of business and popular psychology. Internet research was also used to identify several areas outside the transportation industry and decision science academia that have developed unique and semi-formalized approaches to decision-making that could provide applicable ideas helpful to transportation agency leaders. A particular focus was put on other public-

• sector arenas, since each of these share the challenge of needing to make decisions for the greater good while balancing diverse perspectives.

Transportation Leader Interviews

A core element of the research plan was early and frequent engagement with the target audience of the research: transportation leaders. To make sure the information developed would resonate with the core audience, several focus groups were conducted with transportation leaders in August and September 2023 to provide feedback on the team’s initial content and framework and what they felt was essential for making good decisions in a transportation agency. Focus group participants included the eight individuals identified in Table 3.

Table 3 Transportation agency leader focus group participants

Key insights and takeaways from these early discussions with leaders include:

• The three elements consistently identified as being most important for good decisions included being strongly supported by data and facts, having a good situational awareness, and adhering to the agency’s values. Elements also identified as important but not as strongly include risk assessment, timeliness, open-mindedness, and net-positive results.
• The most important contribution the research could make would be to help transportation leaders establish and apply good decision-making processes.
• Cultivating and maintaining buy-in both leading up to a decision and seeing it through to implementation are the most important components of a decision-making process.
• Groupthink and defaulting to historical approaches are some of the most detrimental elements to strong decision-making at typical transportation agencies.

Initial Research Framework

The core findings of the Literature Review set the foundation for the direction and structure of the rest of the research, which spanned four primary topics that are summarized below. The findings in these areas guided research directions for the tasks that followed.

Part I: Defining a “Good” Decision

Discussions with the NCRHP panel during the meeting to kick-off the project indicated a need to clearly articulate what makes a decision “good.” The research addresses this question first from a broad and universal perspective based on literature and research from the decision science field,

and then explores more detailed answers based largely on discussions with and feedback from transportation leaders.

Part II: Decision Types at Transportation Agencies

Transportation agencies are faced with a wide variety of important and complex decisions. Some decisions occur on a regular cycle while others arrive unexpectedly. To ensure that the research supports the multitude of decision types that occur, this technical memo examines the kinds of decisions that occur regularly at transportation agencies and different ways to classify them. It also highlights important elements of the transportation decision-making context, including the realities of making decisions in a large organization and special considerations for decisions in the public sector.

Part III: How We Make Decisions

Decades of research exist about how people make decisions. This review of the academic and industry literature first outlined the foundations of rational theories of decision-making that dominated thinking for the better part of the 20th century, then pivots to newer theories of decision-making based on empirical observation and study of how decision-making tends to play out in real-world settings. These more recent real-world findings on “non-rational” phenomena serve as the basis for how the advice in the project’s guide is structured and delivered.

Part IV: Resources for Better Decisions

There are a range of formal ways to gather input on stakeholders’ values and priorities, evaluate performance of alternative paths, and ultimately choose a course of action or otherwise make decisions. The team compiled best-in-class technical tools to organize, analyze, and forecast information to support better decisions, which were then assessed in-depth in the Task 2 Toolbox.

Task 2: Decision Toolbox

While the literature review identified universal characteristics of good decisions, the real challenge in any decision situation is to discern what is factual, what the right set of values are, and what practical considerations apply to your situation with acumen. Therefore, the team explored a range of decision-support tools available across industries and disciplines to help leaders and decision-makers achieve these goals. Echoing the focus on facts, values, and practical considerations for good decisions in the literature review, the tools compiled here are organized into three groups:

1. Tools to Get the Facts
2. Tools to Align with Values
3. Tools for Practicality

Within each of these three groups, subgroups tell users more specifically how each tool can help in the decision-making process. Brief summaries on each tool identified are provided with details on what the tool is, when it is most useful, key attributes, high-level steps for implementing it, and resources from which interested decision-makers can find out more.

While all the tools could potentially be applied in DOT decision contexts, the team did not envision including all of them in the resulting Guide or Case Studies. Rather, this Toolbox casts a wide net to explore the kinds of decision-support tools out there that could inform the research approach and key elements for decision-making. The Toolbox was an informative look at the different kinds of technical support available to formalize and begin to quantify decisions, but ultimately the NCHRP panel and executives engaged felt that the Toolbox was likely too technical and detailed for executives to make use of and was not carried forward as a key element of the final products.

Tools to Get the Facts

Being factually competent often means gathering and examining data to ensure you are looking at a decision accurately, but it can also involve gathering and organizing qualitative information.

Tools to Organize Information

• Eisenhower Matrix
• Decision Trees
• Dashboarding
• Affinity Diagrams

Tools to Quantify & Analyze Information

• Cost Benefit Analysis (CBA)
• Correlation Analysis

Tools to Forecast and Predict

• Regression Analysis
• Predictive Management Systems
• Machine Learning

Tools for Collective Input

• Hartnett’s Consensus-Oriented Decision-Making Model
• Modified Borda Count
• Delphi Method (in-depth)
• Stepladder (quick)

Tools to Align with Values

Particularly at a large organization like a transportation agency, multiple staff, partners, and stakeholders will have a say in major decisions. Better leverage their diverse input strategically, build a set of collective values, and figure out which perspectives to prioritize in any given situation with the following.

Tools to Elicit Values

• Multi-Attribute Utility Functions
• Likert scale ratings
• Quantitative Subjective Rating Methods (e.g., Pearson – Tukey)
• Nominal Group Technique (NGT)
• Analytic Hierarchy Process (AHP)
• Analytic Network Process (ANP)

Tools to Prioritize Values

• Voting Schema
• Swing Weighting
• Paired Comparison Analysis
• Simple Multi-Attribute Rating Technique (SMART)
• Preference Ranking Organization Method for Enrichment Evaluations (PROMETHEE)
• Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS)
• Elimination and Choice Translating Reality (ELECTRE)

Tools for Practicality

Decision-making is an intricate and challenging process, especially in the real world where situations are often uncertain and complex. This section presents a collection of tools that can help decision-makers evaluate situations with information that may be limited, incomplete, or imperfect. Common themes include identifying the key factors to consider, the potential consequences of various options, methods to expand perspective, and exploring a wide range of future scenarios beyond the decision itself.

Tools to Assess the Situation

• Vroom-Yetton Decision Model
• Hoy-Tarter Model of Shared Decision-Making
• Strengths, Weaknesses, Opportunities, and Threats (SWOT) Analysis
• Political, economic, social, technological, legal, and environmental (PESTLE) Analysis

Tools to Broaden Your Perspective

• Six Thinking Hats
• Reframing Matrix
• Blindspot Analysis

Tools to Manage Uncertainty

• Game Theory
• Risk Matrix and Tolerance Lines
• Scenario Planning
• Sensitivity and What-If Analysis
• Monte Carlo Simulation (PRA)
• Robust Decision-Making (RDM)
• Futures Wheel

Tools to Optimize Final Decisions

• Greedy - Optimization
• Branch and Bound – Optimization
• Genetic – Optimization

Task 3: Case Studies

The next task was to interview leaders in-depth about a specific decision situation that was particularly challenging or from which they learned valuable lessons. These interviews took place between December 15th, 2023, and June 11th, 2024. These case studies were meant to serve a dual role: as compelling and illustrative stories of decision-making in action to share as a final product in their own right, but also as inputs to the research that the team could “dissect” to better understand key aspects of and strategies for making decisions in challenging situations. Therefore, the case study process began early in the research, starting with identifying who the primary subjects could be.

Case Study Considerations

In thinking about the perspectives that would provide the greatest value to agency leaders, the team identified several considerations to arrive at a robust set of case studies and participants.

• Decision Type: The team wanted examples that would be common enough to be broadly relatable and help with anticipated, recurring decisions. Others should cover unexpected situations to showcase navigating unforeseen situations. In addition, the team wanted to cover examples from across the topical areas identified in the literature review.
• Partner Involvement: Decisions for internal issues require balancing competing interests among departments, which is challenging enough. Decisions involving external partners require an added level of complexity that should be explored.
• Geography: Due to the national reach of NCHRP research, the case studies should represent all major regions of the country as much as possible.
• Participant Diversity: The team intends to use input into the research as an opportunity to ensure that diverse perspectives are embedded in resulting NCHRP products. Therefore, the team also considered diversity in individual gender and race.
• Approach to Decision-Making: Some leaders value facts and data in making decisions, while others pay extra attention to stakeholder buy-in. The stated approach to decision-making was noted by team members with attempts at including a mix of approaches.
• Qualitative Considerations: An individual’s willingness to be reflective, honest, and vulnerable will make a significant difference in the quality of each case study, which was assessed and considered by the team.

The team’s executive advisors proposed 10 industry leaders who could offer breadth and depth on these factors to the NCHRP panel. Following the first Interim Meeting, the list was refined to produce a draft final list of participants to approach about participating.

Case Study Pilot

Before spending the time and resources completing multiple case studies, the team wanted to be sure that the content and format were appropriate for the project. A pilot case study would allow for NCHRP panel review and revision. This pilot was conducted with Vicki Kramer, Director of the Nebraska DOT.

To determine the best situation to use for the case study, the team shared what the goals of the effort were, and the kinds of situations desired from the considerations developed with the NCHRP panel. Director Kramer provided several options based on her experiences, and together she and the team landed on exploring decisions around setting up a new statewide infrastructure hub that crossed industries to help local agencies benefit from newly available federal funds. After a full

interview, the team drafted a sketch of the case study, held a follow up discussion with another staff member, fleshed out the narrative, then shared with Director Kramer for review. The director provided clarifications as well as updates to the situation, which had been in progress at the time of the interview.

From the fleshed-out case study, the team began to pull out some of the notable features of the situation, the barriers it presented, and actions or approaches that were helpful to work through them. These takeaways were pulled out for the team to generalize into preliminary guide content for NCHRP panel review.

Full Set of Case Studies

Once the format and content for the case studies were finalized through the pilot, the team set out to complete a full set of six case studies. The same approach of iterative case study development and culling each case study for relevant content was used for the remaining case studies

A round of in-person interviews took place during TRB’s Annual Meeting with several DOT and agency executives, followed by another round of in-depth virtual discussions to expand the details of the case study’s ideas. In some cases, the team was able to discuss the details of the decision situation with other staff involved. Alternatively, the team conducted online research about the situation to confirm and add details. Draft case studies were shared with the participating executive for review and edits, then revised into final form. The full set of case studies is summarized in Table 2.

Task 4: Interim Findings

An in-person meeting with the NCHRP panel was scheduled for April2024. In advance of this meeting, the team shared notable findings from the research to date and a plan for executing the remainder of the project.

Draft Guide Outline

The team sketched a preliminary outline for the project’s eventual guide based on the combined lessons and insights from the set of case studies. This included the list of challenging issues executives could be likely to face, for which advice was needed. This early outline was circulated to a small group of executives to provide initial feedback and identify other topics to include.

Case Study and Guide Outline Findings

The team shared the combined initial feedback from the case study participants with the NCRHP panel, as well as the feedback from other executives on the preliminary draft of the guide outline. This feedback was summarized in advance of the Interim Meeting to serve as a basis for discussion on the remaining work plan and product development.

Vetting Plan

In order to make sure the guide would be as relevant as possible to its executive audience, the research plan called for early review of a preliminary draft guide by agency executives to ensure key topics were not missed and the content presented was helpful. A plan for this vetting was shared with the NCRHP panel for discussion.

Interim Meeting

This task culminated in the Interim Meeting, in which all above content was presented to the NCRHP panel for discussion and refinement. Key decisions coming out of this meeting included:

• Approving Interim Report 1 and continuation of the work plan
• Refinements to the Draft Guide Outline

• Approval of the Vetting Plan and Proposed Vetters
• Discussion of how to better integrate the Case Studies with the Guide content

Task 5: Preliminary Guide

While lessons and takeaways from each case study provided some content for the guide, the advice gleaned from these discussions did not fill every section. Some topics that the team identified still needed additional content in the form of specific advice and strategies, and the addition of new sections from executive and NCRHP panel feedback on the outline meant that new content was needed. The majority of advice for the guide came from additional interviews with transportation executives, combined with additional research into academic literature and industry resources for select topics.

Interviews

The team embarked on a series of interviews with transportation agency executives who represented a diversity of perspectives and experiences. Armed with a list of topics for the guide, these interviews could be laser focused on getting advice for each decision challenge that would be part of the guide. This gave structure to the interviews and helped to ensure that each topic was sufficiently addressed. The topics for these interviews included:

• Get the Right Information from Your Team
• Monitor Group Dynamics
• Balance Multiple Objectives
• Know When the Information You Have is “Enough”
• Be Savvy to Politics and External Forces
• Embrace Personalities & Internal Forces
• Suspend Protocol Strategically
• Don’t Run from Emotions
• Navigate Crisis
• Adapt Your Decision
• Communicate Your Decision
• Implement Your Decision

Interviewees included:

• Russell McMurry, Georgia DOT
• Nancy Daubenberger, Minnesota DOT
• Joyce Taylor, Maine DOT (Chief Engineer)
• Carlos Braceras, Utah DOT
• Marc Williams, Texas DOT
• Justin Powell, South Carolina DOT

Desk Research

As a real-world, practical guide, most of the topics span lessons hard won by top decision-makers who can share their wisdom if asked the right questions. A few of the topics, however, have a particularly strong effect on even the most seasoned decision-maker, and even these experienced executives may not fully understand why or how they navigate those challenges. These are the topics for which, thankfully, academic research has put recent focus and can provide meaningful answers. These topics include:

• Monitor Group Dynamics
• Don’t Run from Emotions

• Use Mental Shortcuts while Avoiding Biases

The team’s decision science advisor provided resources and direction to add strategies grounded in scientific research for these areas.

Guide Development

The team synthesized all the advice received across these interviews and desk research into the relevant sections, combining similar strategies where needed and placing additional emphasis on those that were brought up by multiple interviewees. A standard format for each topic was developed to include similar subsections, remain within two pages, and incorporate additional content such as quotes, visual elements, and case study references. A limited amount of background information from the project’s literature review was included at the beginning to provide sufficient context to understand the strategies that followed, but care was taken to keep a light touch as the target audience of busy executives would be turned off by long and dense content. This very preliminary version of the guide was pushed to a complete enough state to get meaningful feedback from executives, but it was anticipated to evolve from this initial form.

Guide Vetting

Task 5 culminated with a summary of executives’ reactions to the guide. The preliminary guide was shared with many of the individuals previously engaged

• Russell McMurry, Georgia DOT
• Jack Marchbanks, Ohio DOT
• Nancy Daubenberger, Minnesota DOT
• Joyce Taylor, Maine DOT (Chief Engineer)
• Roger Millar, Washington State DOT
• Carlos Braceras, Utah DOT
• Calvin Reed, Kansas DOT
• Marc Williams, Texas DOT
• Garrett Eucalitto, Connecticut DOT
• Ricardo Martinez, NHTSA (former)

Task 6: Draft Guide and Research Deliverables

Response to the preliminary guide was overwhelmingly positive, so the overall structure and majority of the content remained the same for the next version. Following feedback on the preliminary guide, the team developed a new section requested by reviewers titled Broaden Your Perspective, added an appendix with details about specific biases to be on guard for, and some introductory material on how we make decisions and incorporating values and ethics. Other edits were made throughout, such as changes to terminology, clarifications, or including related ideas in a section.

In addition to the guide, which is the primary product of the research, the team documented methods and findings in this final project report. Initial ideas on how to implement the research findings and get the research into the hands of agency executives are presented in the Implementation Plan. A presentation that summarizes the major findings accompanies the guide and report.

Task 7: Final Guide and Research Deliverables

Final edits to the guide, report, and presentation reflect NCRHP panel comments on the draft versions.