3
Cyber Hard Problems

Chapter 2 explored key factors underpinning difficult cyber challenges. This chapter goes on to list and discuss the list of cyber hard problems—well-defined problems where progress toward their solution would significantly improve the safety and resiliency of cyber and cyber-enabled systems, presented from the perspective of adopters and users.

CYBER HARD PROBLEM 1: RISK ASSESSMENT AND TRUST

• The problem: It is a huge challenge to reliably evaluate the security attributes of a proffered cyber system. There are few predictive security metrics that are reliable, quantifiable, and repeatable. This stands in contrast with simple physical systems—the tensile strength of a rope, for example, can be predicted through modeling and measured by direct testing. Making matters worse, there are readily measured system metrics, such as performance and availability, whose (relatively) simple quantifiable nature can distract from security-related metrics that are difficult to measure. Risk assessments generally need to rely on analysis of components of systems, the architecture according to which they interact, and the choices made regarding the role of the system in operational workflows—all of which influence the extent of system attack surface. An additional consideration is the nature of the threat environment associated with the system and its role; underestimating the

• motivation and capability of threat actors can have disastrous consequences. Information to inform these analyses can be challenging to obtain. And even when it is available, the capacity to perform this analysis in practice, even on relatively small systems, often requires enormous expertise and huge investment. Complicating matters is the non-stochastic nature of cyber risk, unlike more conventional data-informed actuarial risks. Even worse, cyber-risk assessments may feature hidden correlations (such as common components and services deep in supply chains) and secret knowledge (such as threat models and capabilities).¹
• Why it matters: A tiny vulnerability in a seemingly unimportant subsystem can fully compromise the security of a poorly architected cyber system. When the overall system is used in operational contexts where outcomes are consequential and threats are significant, assurances regarding risk are difficult to provide. Risk considerations may therefore drive organizational decisions to limit the scope of application of a system within operational workflows as a means to reduce the potential consequences of attacks. This means missed opportunities to automate functions, improve efficiencies, and create new and significant cyber-enabled capabilities. The use of artificial intelligence (AI) within systems compounds this challenge (as elaborated below).
• What makes it hard: Traditional “black box” testing—testing without being able to see the internal elements of the subsystem—cannot ensure that correct and complete functionality of cyber systems is tested and reduces recognition of cyber vulnerabilities. Extrapolation from even the most carefully chosen test cases is unreliable due to the potential persistent state, nondeterminism in implementations, and hidden logic choices—all hidden within the box. Vendors may resist offering sufficient transparency to evaluators, which means customers and users have to make trust decisions based on judgments made either by third parties or on vendor self-attestation. The norm in vendor licenses is to disclaim liability or warranty of performance, shifting risks upward in the supply chain, ultimately to the end user, who may be in no position to make useful risk assessments—and indeed may be prohibited from doing so due to license terms. Careful auditing and analysis of audit trails can provide some basis for assessment, but systems are seldom equipped with adequate audit logging and effective analysis.
• Where things stand today: Users of cyber systems have to rely on a vendor’s stated commitment to secure development and assessment, perhaps

• augmented by red team assessments that are based on attempts to compromise the system. However, red teaming provides limited coverage, requires expertise, and is expensive. For many system elements, open-source code is often adopted because of transparency, and because users can participate directly in improving the elements. For example, web-based services are based on open-source libraries, such as NPM (node package manager), with millions of available components, making it practically impossible to assess security risks. Indeed, there is now a growing market tension, motivated by open-source issues as well as the recent cyber-focused executive orders (EOs) (such as EO 14028,² advancing a software bill of materials [SBOM] and zero trust) driving incrementally increasing transparency in vendor systems.
• What is needed: Reliable evidence-based assessment by vendors and third parties is needed to drive secure design and facilitate security evaluation. This can be enabled by incentives, such as those proposed in a March 2023 White House Cybersecurity Strategy,³ along with evidence-based external review by third parties and, as noted, graduated degrees of transparency to acceptance evaluation focused on critical security attributes. These incentives can help drive the development of improved capabilities to create and evaluate technical evidence in support of security assurance judgments.

CYBER HARD PROBLEM 2: SECURE DEVELOPMENT

• The problem: Few practices exist today that can reliably ensure that software and firmware system components meet specifications of intended behaviors with respect to security-related quality attributes. Compounding this is the difficulty of developing, in the first place, policy and implementation specifications that relate to cybersecurity and resilience. As a consequence, evaluations focused on security attributes tend to be based on only fragments of direct evidence and, more extensively, on process compliance and other proxies for direct evaluation. These proxy assurances are insufficient in the face of sophisticated attackers. Compounding the problem is the cost and risk of evolving and enhancing systems, which in present practices often requires repeating a full evaluation process.

• Why it matters: When secure development is costly and difficult, system developers are forced to accept limits on capability and complexity for systems that are required to be deployed in consequential circumstances—where there are high standards for safety, security, and adherence to operational rules. Conversely, improvements to secure development practices can open doors—not just to more capable consequential systems but also to more rapid and affordable evolution and improvement of those systems.
• What makes it hard: Improvements for secure development (and evolution) range from tooling and practices to modeling and analysis techniques, language improvements, evidence management techniques, and other technical enablers, as well as skills development for developers and evaluators. The drive for capability and shorter development times—coupled with a perception that secure development is costly, time consuming, and with mostly unmeasurable results—creates counterincentives for development organizations to use secure development techniques. Developing a confidently secured component can entail deep design skills, comprehensive technical knowledge of models and analyses, and attention to details at many levels of design. It also requires, in today’s practice, a base of tools, talent, and practices at a higher level than is typical. This is complicated by challenges of scalability and composability, which require close attention to technical architecture and rules of the road for application programming interfaces and frameworks (see Cyber Hard Problem 3). When hardware and firmware are involved, the challenge is amplified, especially since modern processors create new opportunities for side-channel attacks, among other risks.
• Where things stand today: The lack of effective measures of the various dimensions of quality and security continues to impede the creation of incentives for secure development, and the lack of incentives impedes, in turn, the creation of tools, techniques, and practices for secure development. In organizations where internal measures have been developed, there is broader adoption and advancement of practices, and with good results.^4,5 Some tools to model, analyze, and assure security properties—such as memory safety and safe control flow, for example—are now built into modern programming language designs where these properties “come for free” as language features. With these languages, developers no longer need to use separate tools or

• even to understand the benefits and nuances of the security-relevant property. These properties include type safety (Java, Ada, and more recently Rust, TypeScript, and Hack) and memory safety⁶ (e.g., Rust, Go, and Swift), and an absence of data races (Rust and, as an option, Go⁷). Adoption is extensive because the safer tools enable developers to be more productive, with error warnings earlier in development and, importantly, the new languages offer a near-identical experience to baseline tools. With regard to formal methods more generally, there is now adoption in industrial contexts where, for very specific applications, barriers of scale, usability, and affordability have been successfully overcome. Testing tools, such as fuzz testing, is now widely used because it can offer immediate benefits without wasting developer time on false positives. Chaos testing (roughly, fuzzing at architectural level) is also valuable,⁸ but it requires more elaborate setup.
• What is needed: To have secure development, new engineering practices are needed where engineers can specify and analyze cyber components in a form that provides a measurable high level of confidence in security-relevant quality attributes. When there are practices that have associated outcome metrics, they can inform incentives for producers to use secure practices. The practices can also support composition as a pathway to scale and provide artifacts and evidence that the implementation of these components meets the specification and safety criteria. Experience has shown that techniques are used by developers and evaluators when they yield immediate productivity benefits, are easy to use, and do not require extensive setup and training. Techniques are able to support continuous evolution and evaluation of systems. There are examples where consistency is explicitly managed between the executable elements of larger-scale systems and the body of associated engineering evidence—for example, models, analyses, test cases, inspection information, and operational data. Consistency management of evidence can enable engineers to rapidly and confidently evolve secure systems, since they can more readily reuse evidence—for example, with agile-style practices such as continuous integration/continuous delivery and development, security, and operations.

CYBER HARD PROBLEM 3: SYSTEM COMPOSITION

• The problem: Secure integrated systems are assembled from components and services of varying levels of security by programmers and designers with varying levels of knowledge and understanding, with the intent to achieve an overall secure system design. The goal of “secure composition” is to enable reliance on separately made security judgments regarding particular attributes of the individual system elements (components and services) to support efficient judgments regarding the composite system. A typical modern web application may be composed of hundreds of components drawn, for example, from the NPM open-source ecosystem, which offers more than a million components. These components can themselves be complex and composed of smaller separate components, with complex interdependencies. Another example is the architectural pattern of micro-services, which enables a more modular and scalable approach to operations on shared data. Individual examples notwithstanding, there is no comprehensive science of safe composition to guide integration, nor are there generally usable tools to validate compliance with semantic rules that allow safe composition.
• Why it matters: Within a safe composition framework, efforts to support security evaluations can be “reused” when the evaluated components and services are combined into diverse systems (assuming composition rules are followed). Without composition rules, which is more often the case, complex systems can be infeasible to evaluate, especially when they include complex and opaque components and services.
• What makes it hard: Success in composition can depend on deep technical properties of both the components to be composed and the technical design rules according to which the compositions are constructed. These properties and rules may be specific to particular security attributes, so there can be significant difficulty in achieving aggregate security judgments of fitness for use. On the other hand, and very importantly, incremental progress can be made, attribute by attribute.
• Where things stand today: Development of safe system composition frameworks depends mainly on talented designers and developers, as well as deep knowledge of components. But many components and services are available through development environments and repositories (such as GitHub) that can assist developers in assembling and curating components and performing some level of continuous system testing. Some of the most useful composition attributes are “hidden” in the designs of programming

• languages and runtime systems—for example, “type safe” programming languages enable components to be combined by a linker into overall systems that are type safe as a consequence of the type safety of the constituent components.
• What is needed: Principles and architectures for system composition, advancement of expert practice in integration design, and more effective tools to support integration activities. A key enabler of composition and scale is architectural design that minimizes interactions among system elements and that enables resilience—graceful degradation in the event of compromise of a system component. That is, functions are partitioned so that an error in one partition does not adversely affect other partitions. This becomes increasingly significant with the growing scale and interconnections of systems and supply chains. Importantly, the capacity to effectively organize systems for resilience has to be planned early in development and is difficult in practice. It is also difficult to assess likely resilience outcomes early in the design process when architectural decisions are made. Chaos testing addresses this, but it occurs late in the process after systems can be tested. Resilience and composition are important areas for research focus, since “bolting on” resilience late in an engineering process is, arguably, even more challenging than bolting on other aspects of security.

CYBER HARD PROBLEM 4: SUPPLY CHAIN

• The problem: System elements for complex integrated systems can be sourced by a diverse array of suppliers, with components and libraries sourced from vendors, open-source projects, and custom software developers, and services sourced from diverse cloud and software-as-a-service offerings. Within subordinate supply chains, these suppliers may operate under different government or industry rules. It can be hugely challenging for an integrating system designer to readily and safely leverage the diversity of components and services required for system engineering projects, whether they are bespoke national security systems or simple commercial web applications. Supply-chain challenges are compounded by the technical challenges of composition and architecture-derived resilience.
• Why it matters: The diversely sourced system elements all interact within the architectural framing of the integrated system. Even when they are not opaque to analysis, typical for commercial components and services, the

• interactions are difficult to model and predict. This can stymie confident acceptance evaluation, and in many cases limit the range of operational contexts within which the integrated system can be safely operated. Managing these interactions is analogous to achieving a kind of zero trust at every layer of system design, from hardware level up to major subsystems.
• What makes it hard: Commercial suppliers, in the interests of protecting trade secrets, generally do not want to reveal details of their components, either to end customers or to other participants in the supply chain for an integrated system. Complicating matters is that most systems employ large numbers of diverse system elements from a vast array of suppliers, commercial and open source, including both components and services. Without some degree of transparency, however, it is not possible to make sound assurance judgments at any stage in a supply chain. In many cases, transparency is provided but limited to trusted third parties or intermediaries. Even in this case, components are rarely specified in the detail required for secure composition and may only occasionally be updated or improved to address vulnerabilities as they become known. When updates are made, however, often extensive testing is required to ensure that repairs and enhancements are fully compatible with existing system elements. In the absence of updates, however, insecure components may need to be encapsulated in ways that protect other parts of the system should those components be compromised.
• Where things stand today: There is a movement to require end products to include an SBOM and, eventually, a hardware bill of materials. EO 14028 enshrines this intent for government systems.⁹ However, SBOM data give only a hint of potential security issues in an integrated system. Open-source software provides a wide array of components that can, in principle, be directly evaluated and whose cost to maintain, evaluate, and improve is shared, but relatively few open-source projects enjoy a sufficient level of attention to code changes to assure continuing safety.
• What is needed: A reliable supply chain is a mechanism, supported by architectural commitments, to assure that integrated systems developers can be confident regarding diversely sourced components and services. One step along this path is for architectural decisions—at every level of design—to isolate and minimize privilege for system elements that cannot be readily evaluated or vouched for. Another step along this path is to select and curate critical open-source components that can meet the secure development desiderata. This may include investing in augmenting evidence creation in

• the open-source project to support more confident and efficient security judgments. Some industry segments, such as the automotive industry, have started to define criteria for hardware and software components as a step along this path.

CYBER HARD PROBLEM 5: POLICY ESTABLISHING APPROPRIATE ECONOMIC INCENTIVES

• The problem: The suppliers of cyber systems are seldom held liable even for the shoddiest products. Nor are there sufficient rewards for high quality. The well-known lemon law of economics explains why opaque cyber systems often manifest poor security as the norm. Occasionally, “brand” and informed customer demand can drive desired behavior in manifesting quality. Suppliers have avoided effective regulation with the argument that increased accountability would hinder innovation and national competitive advantage—and with an additional argument that security attributes are hard to measure, even for original developers, and therefore hard to vouch for. Many policy initiatives lead to process-focused mandates with self-attestation of compliance. These are weak proxies for delivered security in products and services. As a result, incentives have been and continue to be misaligned, leading to a market failure regarding accountable security.
• Why it matters: Cyber systems remain insecure, despite considerable research and attention to security practices. Consequences are experienced almost universally, from individual consumers and firms to national security applications and civil infrastructure. Even an explicitly incremental approach, taking small steps, could make a significant difference in security outcomes.
• What makes it hard: The complexity of cyber systems makes it technically difficult, and often infeasible, for even the best-intentioned supplier (or insurer) to warrant a system as free from important categories of defects or vulnerabilities. There is a “catch 22” situation where the inadequacy of security practices and tools impairs potential for warranting results—but without this potential, there is less incentive to invest in advancing those practices and tools. This is a measurement paradox, where the inability to directly assess levels of security in products and services can impair investment in practices to assure security—and also in ways to better assess security.
• Where things stand today: There are proposals, including in the 2023 White House National Cybersecurity Strategy, to create liability exposures for

• vendors for certain security defects under tort law.^10,11 There are significant technical challenges. For example, the rapid evolution of systems and engineering practices would not align well with statically defined bars for liability. Additionally, the expense of litigation can often be an insurmountable barrier to be a useful remedy for most users. The bar for activating tort liability would need to be low if there are to be clear rules favoring efficient litigation. Vendors have encouraged the development of self-attested compliance to process standards as a substitute, but such compliance is not always an effective predictor of security outcomes and can be gamed by sophisticated suppliers. Additionally, compliance standards themselves can be a barrier to innovation and competition, since they focus on process rather than outcomes.
• What is needed: Economic incentives and transparency that can facilitate measurement and thereby enable reward for actual security, with consequences for poor security. From a legal perspective, this could include a concept of reasonable care in development based on evolving improvements in technology and, in the case of suppliers of long-lived and scalable systems, continuous improvement. The trouble is that it is difficult to provide an adequate and effective commercial legal definition of “reasonable care.” Societal and commercial issues are diverse and contentious, which complicates the development of solutions.

CYBER HARD PROBLEM 6: HUMAN–SYSTEM INTERACTIONS

• The problem: The human interfaces presented by cyber systems are often confusing and uninformative, even to expert operators and users, and can even encourage unsafe behavior because they are poorly suited to purpose. This includes many kinds of security-related interactions, ranging from authentication and privacy control to configuration of access policies and responding to security alerts. Inadequate attention to user design for security and privacy (coupled with a failure of market incentives and regulation) has led to growing problems establishing or maintaining user security and privacy. Compounding this is the complexity and nuance of many security-related interactions between systems and humans. Indeed, errors in human interaction are by far the dominant cause of security breaches.

• Why it matters: System engineers may have optimistic attitudes regarding the potential for training and awareness of human operators and users as a principal means to mitigate risks posed when attackers attempt to exploit human weaknesses and vulnerabilities.^12,13 The ongoing success of phishing and social engineering attacks illustrates the lack of success of this approach. In many circumstances, the mitigations most likely to make a difference are design adaptations to the human interface, including judicious choices regarding operational workflow structure and interaction design.
• What makes it hard: The committee sees, on the one hand, broad benefit for users to oversee decisions affecting the security of their activities and data. But, on the other hand, it can be challenging in system design to afford human operators and users the ability to make those decisions in an informed and efficient manner. In the specific case of privacy policies,¹⁴ for example, a study found that it would take an average user 30 full working days were they to read the policies for the sites they visit over the course of a year, and presumably even more time to understand how these policies interact with each other, and the useability and usefulness situation has not improved more recently.¹⁵ For web-browser plug-ins or other downloaded programs, there is less information available, and poor choices can lead to intrusions such as ransomware attacks. From a technology perspective, long-understood security principles, such as least privilege and auditability, are often not sufficiently respected in frameworks for browser extensions and mobile apps.
• Where things stand today: User studies have had a significant role in informing human-interaction design, such as in 2017 when the National Institute of Standards and Technology offered a dramatic turnabout on password guidance,^16,17 shifting from system administrator folklore regarding password construction rules to science-informed guidance. In this and other

• aspects of human interaction, system engineers are starting to understand that there is not an immutable trade-off between security and usability, and that attention to good human-engineering practices, informed by empirical science, can lead to systems that are both more secure and usable.
• What is needed: The science of user-focused secure design needs to go beyond the focus on the user’s primary functional task to encompass and enable user security with varied applications and supporting policies for data governance to enable better privacy protection.^18,19 This needs to include developing an informed understanding of the limits of human capacity to consistently adapt behavior through guidance and training. Beyond these limits, the only means to enhance security is to adapt the engineering of systems and associated workflows. An important additional element of this design challenge is the development of means by which users and operators can more readily monitor, understand, and regulate information captured and shared on their systems. A particular challenge is how to achieve this without producing information overload, introducing inefficiency, and, through this transparency, creating new avenues for attack.

CYBER HARD PROBLEM 7: INFORMATION PROVENANCE, SOCIAL MEDIA, AND DISINFORMATION

• The problem: Social media platforms are complex cyber systems that are used by millions of people and that can profoundly affect opinions across broad populations. These platforms provide tools that not only create social connection but also enable third parties to achieve precision targeting at scale to manipulate not only opinions but also perpetrate scams. These actions can be achieved through designed features of the business model and through use of data and deceptions to spoof content policies and controls, including deep fakes. Both aspects of this problem pose complex technical and, as widely reported, policy challenges. An additional challenge is user privacy, since online media gather highly granular usage information to inform the algorithms that tailor content to user interests and that direct advertising content.

• Why it matters: With proliferating deep fakes, the potential trustworthiness and reliability of online media are increasingly threatened. Without focused effort, the war of attrition between production and detection of fakes could fail. Additionally, the data gathering done by social media platforms to inform algorithms (and AI model training) can be highly granular and revelatory, with obvious privacy concerns, as evident in policy discussions surrounding TikTok.
• What makes it hard: The capacity for modern AI to produce deep fakes is rapidly advancing, as is the capacity to detect fakes. It is unclear whether detection capabilities can keep up. Adding to the asymmetry is the tempo of operations, which, in online media, can be very fast. Misinformation can rapidly proliferate and be precisely targeted, at which point it becomes very difficult to counter it. Also advancing, however, is the ability to watermark legitimate content such as images and videos. There are some indications that social media users can be assisted to be alert to misinformation, but this may not keep pace with the capability of producers. On the policy side, attempts to hold platform providers liable are often prevented on the basis of free speech rights and with Section 230 of the Communications Decency Act of 1996, an absence of legal incentives and, in some cases, a business model that creates counter incentives. Platforms have business incentives to maximize “engagement,” which in the absence of safeguards can occur even when algorithms amplify and reinforce adverse messaging.
• Where things stand today: There are also potential means for image and video producers to inhibit deep-fake production using watermarks and other techniques. There is little legal liability, especially in the United States, for knowingly disseminating false information on social media. Data privacy has more policy safeguards, particularly in the European Community with the General Data Protection Regulation, but the technical security challenges remain.
• What is needed: On the policy side, disincentives could be enhanced for knowingly disseminating false information, and accountability could be increased regarding protection of personal data gathered in support of algorithms and advertising placement. Techniques for detecting deep fakes and watermarking original content require continual enhancement in the face of the rapid progress in generating and disseminating deep fakes, including imagery, video, voice, and other modalities.

CYBER HARD PROBLEM 8: CYBER-PHYSICAL SYSTEMS AND OPERATIONAL TECHNOLOGY

• The problem: Cyber-physical systems (CPS) include both operational technology—used, for example, in manufacturing, civil infrastructure, and transportation—and Internet of Things technology used in telecommunications, network hubs, security cameras, smart locks, thermostats, televisions, home, and industrial controls. CPS encompasses computing software and hardware, as well as sensors and actuators that interact with the physical world. CPS are both vulnerable and consequential. They are vulnerable because, historically, they had generally been presumed to be “off the net,” and so inherited an engineering tradition with a lower standard of security than information technology (IT) systems. Additionally, CPS often lack access pathways for updates, and, finally, they can offer avenues of attack via audio, network, and physical access.
• Why it matters: CPS are highly consequential because they are the control fabric for civil infrastructure systems, industrial controls, and manufacturing systems, as well as embedded national security systems. They are also now critical infrastructure in homes and offices. There have been many significant attacks over the years in nearly all of these sectors. That they are largely invisible, as if part of the furniture, often causes them to be ignored.
• What makes it hard: Secure design of a CPS requires both hardware and software expertise. CPS deployments generally do not have access to trained professionals prepared to work on these systems, and many do not support remote update should repairs need to be made or vulnerabilities patched. This means that the technical challenges of reengineering hardware and software for security are compounded by the high stakes of “getting it right” at the outset. Additionally, many of these systems feature real-time controls, which can create a tangle of internal interdependencies that complicates reengineering for security and resilience, both for individual CPS devices and at scale for distributed networks.
• Where things stand today: There is technical attention to CPS security, but challenges remain to develop real-time systems that are more modular, scalable, and secure.
• What is needed: CPS are important targets, and consequently require all the protections of modern IT systems—as well as the requirements previously noted regarding transparency, configuration integrity, critical evaluation, secure design and composition, and secure supply-chain requirements. Specific focus

• on CPS is important because many of them provide real-time control, which leads to both design and assurance challenges. Engineering practices are also an issue, because many embedded systems need support for rapid remediation of issues, including critical infrastructure and weapons systems.

CYBER HARD PROBLEM 9: ARTIFICIAL INTELLIGENCE AND EMERGING CAPABILITIES

• The problem: Contemporary AI is at the leading edge of a technology storm for which it may be difficult to predict the full set of safety and security ramifications. Today, AI is generally based on neural-network models supporting machine learning (ML) and generative AI using statistical inference based on large corpora of training data. Despite widespread adoption, protection criteria for modern AI technologies and systems are still nascent.²⁰ Efficacy is dependent on many factors including the reliability of training data, but these data are often opportunistically acquired and poorly curated, biased, and inaccurate. Dependence on training data introduces complex issues beyond quality and security, including copyright and data ownership. Another factor is that even sophisticated users may attribute characteristics to modern AI systems based on sampled experience, which may not reveal the diverse kinds of weaknesses and vulnerabilities that are often present. This may become problematic from both a security and safety perspective as traditionally human-centric tasks are automated by the use of AI. (That AI can be used to automate offensive cyberoperations is addressed separately in the section “Defense Against Offensive Artificial Intelligence” in Chapter 4.)
• Why it matters: Modern AI models are becoming significant elements of a growing range of systems. AI has a wide range of current applications and an enormous range of potential applications. Although these include some safety critical applications such as transportation and medical image analysis, the AI systems are generally in advisory roles where accountability resides with human operators. But there are aspirations to apply modern AI in contexts where trustworthiness is essential, including faster-than-thought autonomy, technical systems engineering, and a range of expert applications.
• What makes it hard: Security depends on a carefully developed, principled framework where weaknesses and vulnerabilities can be identified

• and clearly tied to potential adverse outcomes. In ML, a vast corpus of training data, often undifferentiated with respect to quality and correctness, are transformed into neural logic in the form of up to hundreds of billions of parameters in a neural network. This creates an opacity to analysis such that, even with full transparency of the system and its parameters, the behaviors of neural networks cannot be reliably predicted. Even with carefully curated training data, the statistical nature of neural-network models means that outputs, whether from ML models or generative AI models, can be inexact and untrustworthy. A statement “the neural network will never do this” is often impossible to assure. Thus, the human factor compounds the difficulty of the problem due to automation bias—the propensity for humans to favor decisions from autonomous systems. The security of the applications also resurfaces old security challenges in this new paradigm. For example, applications leveraging generative AI may be susceptible to SQL-like injections from untrusted sources (indirect prompt injection) because they leverage the same input pathways for a neural network for both “data” (untrusted documents) and “code” (textual instructions), violating a core security principle. Making matters worse is the extraordinary breadth of attack surface, encompassing training data, network architecture, and operational inputs—all in addition to traditional cybersecurity attack surfaces associated with AI models as software components and services.
• Where things stand today: In the familiar “opportunity and challenge” framing, modern AI systems are extreme in both their great promise and great potential peril. Current generative AI systems can provide both amazing insights, due to their extraordinary powers of recall over the vast ocean of their training data and information sources provided at runtime, and also laughably incorrect hallucinatory conclusions, due to the fundamentally approximate representation of the “knowledge” embodied within a network (including falsehoods, irony, and AI outputs), and the stochastic nature of response generation. There are, however, a wide range of advisory applications where modern AI systems work very well. Adaptations to both the systems (e.g., plug-ins, retrieval augmented generation, agentic systems, and similar techniques) and the operational workflows that incorporate them (e.g., protocols for human operators and guidance for human users) enable systems developers to exploit the strengths of these models. It is, however, the early days for AI, and there are no agreed-upon security foundations.
• What is needed: As a rapidly evolving automation technology that is powerful but not easily predictable, software developers that leverage AI must place

• explicit human accountability and safe human outcomes as primary design priorities. There are many steps that can be taken with modern neural-network models to make incremental improvements, but it is important to note that many of the weaknesses and vulnerabilities are intrinsic to the neural-network technical architecture. As such, foundational improvements in AI modeling or model augmentation are needed to make models more auditable for reliable use as system components. Paths forward may include the hybrid use of neural networks with symbolic techniques, wherein models are augmented with explicit knowledge graphs and logic-based deductive processes. Since at present, models themselves are not auditable, it is important, not just to identify weaknesses and vulnerabilities in modern AI, but to develop a principled security practice that includes modeling and analysis techniques for detection and mitigation. Applying secure design principles (e.g., access controls, trust boundaries, and data/control separation) may remediate many security and safety challenges with generative AI applications, but appropriate incentives are apparently still required and frameworks for developing generative AI systems that explicitly promote or enforce them. Lastly, as general-purpose knowledge systems that often include generic built-in safety and security guardrails, improvement to application-specific safety and security guardrails is required. This should include supplementary external guardrails independent from those trained explicitly into a model so that controls can be scoped and enforced independently and provide capability parity to the generative model.²¹

CYBER HARD PROBLEM 10: OPERATIONAL SECURITY

• The problem: There is increasingly pervasive dependence on large-scale systems such as cloud infrastructure from Amazon Web Services, Azure, and Google, as well as scaled applications such as search and email from Microsoft and Google. For these large-scale systems, users have little insight, beyond vendor self-attestation, into the state of security configuration. These examples are of mainstream consumer applications, but similar considerations arise within the IT infrastructure of large organizations including, for example, enterprise resource planning systems.

• Why it matters: As systems grow in scale, interconnection, and significance to their users, resilience becomes increasingly important, with concerns over confidentiality, integrity, and availability for both data and operations. Resilience to attack and failure can include techniques to isolate compromised system elements to enable the system to continue to operate through the compromise, analyze the situation, and subsequently recover. Defined processes for operational situational awareness and remediation are increasingly critical for practical operational security as well as assessment, both real time and forensic. This is not possible unless attacks can be detected quickly, potential consequences assessed rapidly, and defensive actions initiated appropriately.
• What makes it hard: Operational security for larger organizations poses multiple challenges—prevention, detection, response, and recovery. Anticipation and mitigation mean reducing both potential for successful attacks and also the extent of consequences when attacks (and failures) result in compromises. These can involve significant planning and cost. Detection and response, for example, require threat intelligence, constant practice, the diligent development of supporting tools, and comprehensive knowledge of overall organizational networks and applications. Detecting attacks and determining consequences can be expensive and error prone. Many government systems, for example, include diverse elements from diverse vendors, including security support. Recovery from attacks is an important element of security and, as has been shown by numerous ransomware attacks, many organizations struggle to accomplish this. Additionally, many societal systems depend on a few key infrastructures, such as cloud-based services, which can become single points of catastrophic failure, with risks of compromise, availability, and integrity. High levels of operational security require organizational maturity and top-tier support due to costs, complexity, and the need for architectural control. Business and market incentives pose challenges that are amplified by the difficulty of assessing capacity with respect to the several dimensions of operational security.
• Where things stand today: There are large organizations that have developed well-tuned processes, continuous monitoring and assessment, as well as effective tools to manage at-scale systems (and complex configurations) that include engineered reliability. In many cases, this knowledge is closely held and difficult to assess from the outside. Operational security for smaller organizations (and individuals) remains hit or miss.
• What is needed: Systems need to be designed to support and anticipate operational security needs, including detection of attacks via automation and

• automatic assessment of the potential consequences of attacks and remediation actions. Despite the typically unique organization-specific designs of large operational systems, there are common principles of design and operations that can be identified, applied, and measured.

Readers will easily recognize that these hard problems are not independent of one another and cannot be solved individually. Therefore, in highlighting them, the committee also hopes that collective action can be organized across government, industry, and research communities to make progress addressing them.