New Threats Arising from AI System Deployment and Use

The preceding sections examined how AI systems are reshaping offensive and defensive cybersecurity capabilities. A separate but equally important set of challenges arises from the use of AI systems as components of operational environments. As AI systems increasingly replace or augment conventional software and human decision making, they introduce new classes of threats that differ in important ways from those associated with traditional computing systems.

From Deterministic Programs to Probabilistic Systems

The rules and logic in conventional programs are largely human generated. Programs are written with a clear intent, the intended behavior can be documented, and unexpected outputs are typically classified as bugs that can be identified and corrected. By contrast, the behavior of generative AI systems is established largely through training rather than explicit specification. As noted earlier, although training and fine-tuning can reduce undesirable behaviors, they cannot eliminate them entirely, and model builders can generally provide only probabilistic rather than deterministic assurances about system behavior (Wei et al. 2022; Naseem 2026). This distinction introduces new security challenges, particularly when AI systems are deployed in operational environments where unexpected behavior may have consequential effects.

Adversarial Inputs and System Behavior

AI systems are often deployed in environments where adversaries can deliberately craft inputs to provoke unwanted behavior. Techniques such as prompt injection and other forms of input manipulation can cause systems to behave in unintended ways, potentially bypassing safeguards, exposing sensitive information or triggering harmful outputs. In agentic and tool-using systems, the risk is especially pronounced because malicious instructions may be embedded not only in text prompts, but also in images, metadata, retrieved documents, or agent-to-agent messages, allowing attacks to propagate across a broader system graph (Syed et al. 2025).

Achieving meaningful testing coverage for adversarial inputs is difficult owing to the enormous state space available to attackers. As a result, vulnerabilities may persist even after extensive evaluation, especially when systems operate across multiple modalities, tools, and external data sources. Prompt injection risk is not confined to the model itself, but emerges from the interaction between inputs, memory, retrieval, and autonomous workflows, making mitigation an ongoing and iterative process rather than a one-time fix.

Human Interaction and System Misuse

Human interaction with AI systems creates new classes of threats. Conversational systems typically produce outputs without conveying calibrated confidence, which can lead users to accept incorrect or misleading information. In some cases, systems may generate plausible but incorrect responses (hallucinations) or be manipulated through carefully crafted inputs (jailbreaking).

Such manipulations can exploit weaknesses in input filtering and safety mechanisms. For example, prompts that use indirect phrasing, non-English language, or encoded formats such as base-64 may bypass safeguards intended to restrict harmful behavior (Jiang et al. 2024). These techniques can induce systems to produce outputs that violate intended constraints, including generating insecure code, revealing sensitive information, or providing guidance that could be used to carry out harmful activities.

In cybersecurity contexts, this problem has more direct operational consequences. A variant of hallucination arises when AI systems are used to generate computer programs or security-relevant code. These outputs may appear correct and functional, but can include subtle errors, insecure patterns, or unintended behaviors that are difficult to detect—particularly for less experienced users. In practice, this may result in the introduction of new vulnerabilities into systems that are intended to improve security.

These risks are compounded by user behavior. Overreliance on AI systems, particularly when outputs appear confident or authoritative, may reduce scrutiny and increase the likelihood that errors go undetected. As AI-generated outputs are incorporated into operational workflows or deployed at scale, incorrect assumptions embedded in code, configurations, or analysis may propagate through systems before being identified, increasing the potential impact of failures.

Data Access and Information Exposure

Many AI systems are deployed with access to external data sources, including sensitive or proprietary information. Retrieval-augmented generation (RAG) systems connect LLMs to data stores such as enterprise documents, communication systems, or databases. While these architectures enable more useful and context-aware outputs, they also introduce new risks.

The most consequential security implication is that users interacting with such systems may gain indirect access to underlying confidential data. In practice, AI systems may not reliably enforce data access controls or respect sensitivity boundaries, particularly when operating across heterogeneous inputs. As a result, sensitive information may be exposed through model outputs, even when direct access to the underlying data would otherwise be restricted.

To mitigate this risk, deployments could avoid allowing users to interact with a RAG system if they would not otherwise be permitted to access the underlying data. System architects can instead partition their deployments so different subsets of authorized users interact with systems tailored to their respective data access requirements. More broadly, system designers should not assume that LLMs will reliably honor data sensitivity or compartmentalization constraints.

These mitigations, however, are not always feasible or desirable in practice. For example, customer-assistance systems often require access to internal knowledge bases, including frequently asked questions, known issues, and operational workarounds. While organizations may not wish to fully expose such data, the business value of improved functionality and user experience may outweigh the risks of partial information exposure. This creates a trade-off between utility and security that must be explicitly managed, rather than assumed away through system design alone.

Agentic Systems and Autonomous Action

As discussed earlier, increasing autonomy is a defining characteristic of emerging agentic systems. These agentic systems can use external tools, execute multi-step workflows, and adapt to changing conditions, sometimes with limited or no human oversight. Simple chat systems can function as sophisticated assistants, but there is typically a human in the loop to make a final judgement call. By contrast, tool-equipped AI systems may perform actions with financial, safety, and security implications, introducing new risks as autonomy increases.

The security implications of this change are profound. Rather than eliminating asymmetry, agentic AI may deepen it by expanding the combinatorial space of possible attacks faster than defensive systems can adapt (Li and Zhu 2025). These systems may act on ambiguous instructions or adversarial inputs embedded within otherwise benign data sources, leading to unintended or harmful actions. A growing ecosystem of third-party and custom agents, along with emerging standards for tool integration, is expanding the range of systems that can be accessed and modified through AI. As more programs and services become agent accessible, AI systems may be able to perform actions comparable to those of authorized users, increasing both capability and risk.

One challenge is that AI systems are often instructed in natural language, which can be ambiguous, underspecified, or misinterpreted (Thornton 2026). This is further compounded by the expectation that the AI system will process potentially adversarial inputs, even when acting on behalf of a legitimate user. For example, a request such as "Send me an email summarizing action items from my inbox last week" may involve processing content from an attacker who sent an email that attempts to override the user's intent. Because these systems must infer intent rather than execute fixed logic, they are more susceptible to manipulation, particularly in environments where inputs cannot be fully trusted (Howard 2019). As the range of actions available to these systems expands, so too does the potential impact of errors or exploitation.

Agentic AI security is an active area of research. Existing access control systems have been effective in traditional settings because they rely on well-defined policies enforced by deterministic mechanisms developed over decades. It remains uncertain whether these approaches can be adapted and extended to systems that interpret natural language instructions and operate in adversarial environments, or whether new architectures—potentially involving AI systems that monitor and constrain other AI systems—will be required. Active research is exploring structural and deterministic controls over agent access, identity, action constraint, blast radius, information flow, and auditability. Mitigating these risks will likely require layered safeguards, including stricter validation of actions, constraints on high-risk operations, and continuous monitoring to detect and respond to anomalous behavior.