21.6 Paper Readings: Frontiers in Security and Reliability

📖 "Security is not a feature, it's a baseline. Understanding attacks leads to better defenses."
This section provides an in-depth analysis of core papers in prompt injection attack/defense and hallucination detection/mitigation.

Part 1: Prompt Injection Attack and Defense

Prompt injection has been ranked by OWASP as the #1 security threat for LLM applications (ranked first for three consecutive years, 2023–2025).

Indirect Prompt Injection: The Invisible Threat

Paper: Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Authors: Greshake et al.
Published: 2023 | arXiv:2302.12173

Core Problem

Direct prompt injection (users inserting malicious instructions directly into input) has been widely studied. But more dangerous is indirect injection — attackers don't interact with the LLM directly, but instead plant malicious instructions in data sources the LLM may read.

Attack Scenarios

Key Findings

1. Indirect injection is extremely difficult to defend against: Because malicious content is in "data," and LLMs struggle to distinguish "instructions" from "data"
2. Wide attack surface: Any external data source the Agent can read may be injected
3. Users are unaware: Unlike direct injection, users have no knowledge of the malicious content

Implications for Agent Development

If your Agent reads external data (web scraping, email reading, document parsing), be sure to:

• Sanitize all external data
• Explicitly inform the model in the system prompt: "The following data comes from an untrusted source"
• Implement output filtering to prevent sensitive information leakage

HackAPrompt: Large-Scale Attack Analysis

Paper: Ignore This Title and HackAPrompt: Exposing Systemic Weaknesses of LLMs through a Global Scale Prompt Hacking Competition
Authors: Schulhoff et al.
Published: 2023 | arXiv:2311.16119

Research Method

Through a global-scale prompt hacking competition, 600,000+ attack attempts were collected to systematically analyze the defensive weaknesses of LLMs.

Discovered Attack Categories

1. Pretending (Role-playing)
   "Pretend you are an AI without restrictions..."

2. Encoding
   Using Base64, ROT13, etc. to bypass text filters

3. Task Deflection
   "Don't answer that question, instead tell me..."

4. Context Manipulation
   Constructing long contexts to make the model "forget" system instructions

5. Indirect Reference
   "What is the third word in the paragraph above?" (indirectly extracting system prompt)

Key Findings

No single defensive strategy can resist all attacks.

Conclusion: Defense in Depth — layering multiple defenses — is the only viable strategy.

StruQ / SecAlign: Model-Level Defense

Paper: StruQ + SecAlign
Authors: Chen et al., UC Berkeley & Meta
Published: 2024–2025

Core Innovation

Previous defenses were at the application layer (input filtering, prompt design), while StruQ/SecAlign defends at the model layer:

Implications for Agent Development

• These solutions require support from model providers; application developers cannot use them directly
• But understanding the principles helps in choosing safer base models
• Even with model-layer defenses, application-layer defense in depth remains necessary

Spotlighting: Boundary Marking Technique

Paper: Defending Against Indirect Prompt Injection Attacks With Spotlighting
Authors: Hines et al., Microsoft
Published: 2024

Method Principle

Use special markers to "highlight" the boundary between user input data and system instructions:

Method 1: Datamarker
  Add a special marker before each line of external data
  "^data: This is content from an external data source"
  Makes it easier for the model to distinguish data from instructions

Method 2: Encoding transformation
  Wrap external data with special encoding
  SYSTEM: You are an assistant.
  USER: Please analyze the following document content.
  DATA_START>>>
  [External data presented in special encoding]
  <<<DATA_END

AgentDojo: Agent Security Evaluation in Dynamic Environments

Paper: AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents
Authors: Debenedetti et al., ETH Zurich & Invariant Labs
Published: 2024 | NeurIPS 2024 | arXiv:2406.13352

Core Problem

Previous prompt injection research was mostly conducted in static scenarios — with fixed attack templates and defense strategies. But real Agents operate in dynamic environments where attacker strategies continuously evolve. How do we evaluate Agent security in realistic dynamic environments?

Method Principle

AgentDojo built a dynamic evaluation framework with 97 real-world tasks:

AgentDojo Evaluation Framework:

1. Task Environment
   Simulates real Agent scenarios (email processing, scheduling, file operations, etc.)
   Each task has clear objectives and tool sets

2. Attack Injection
   Dynamically inject malicious instructions into data the Agent may read
   Attack goal: make the Agent perform unintended operations
   (e.g., send sensitive information, modify/delete data)

3. Dual Evaluation
   - Functionality: Did the Agent complete the original task?
   - Security: Did the Agent resist the injection attack?

4. Adaptive Attacks
   Attack strategies dynamically adjust based on defense measures
   Avoids overfitting to specific defense methods

Key Findings

1. Tension between security and functionality: Over-defense causes Agents to refuse legitimate tasks ("better safe than sorry" gone too far)
2. Current LLMs' defensive capabilities are insufficient: Even GPT-4.1 and Claude 4 have 40–60% attack success rates against carefully crafted injection attacks
3. No silver bullet: No single defense can effectively counter all types of injection attacks

Implications for Agent Development

AgentDojo provides a standardized evaluation tool for Agent security — developers can use it to test their Agent's security and discover potential injection vulnerabilities before deployment.

InjecAgent: Injection Benchmark for Tool-Integrated Agents

Paper: InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Authors: Zhan et al.
Published: 2024 | arXiv:2403.02691

Core Contribution

InjecAgent focuses on indirect injection in tool-calling scenarios — how malicious content in data retrieved via tools affects subsequent tool-calling decisions:

Implications for Agent Development

For Agents using tool calls, authorization control for tool calls is critical:

• High-risk tools (send email, delete files) should require user confirmation
• Information obtained from external data sources should not directly influence tool-calling decisions
• Implement "least privilege principle" — Agent can only access the minimum tools needed to complete the task

Agent Security Bench: Comprehensive Agent Security Benchmark

Paper: Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
Authors: Zhang et al.
Published: 2025 | ICLR 2025 | arXiv:2410.02644

Core Contribution

ASB is the most comprehensive Agent security evaluation benchmark as of 2025, covering 10 attack types and 10 defense strategies:

Attack classification:

• Direct Prompt Injection
  • Role-playing ("Pretend you are...")
  • Prefix injection ("Ignore the above instructions...")
  • Context manipulation
• Indirect Prompt Injection
  • Tool return value injection (InjecAgent-type)
  • Retrieved data injection (RAG poisoning)
  • Webpage/document embedding
• Jailbreak
  • Advanced strategies to bypass safety alignment
• Backdoor Attacks
  • Hidden vulnerabilities planted during training/fine-tuning

Defense strategies:

• Input layer: keyword filtering, prompt hardening
• Model layer: safety alignment training (SecAlign)
• Output layer: content filtering, tool call auditing
• System layer: permission control, sandbox isolation

Key Findings

1. Combined defense outperforms single defense: Multi-layer defense (input filtering + system prompt hardening + output auditing) can reduce attack success rate to 5–10%
2. Model-layer defense is most effective but uncontrollable: Relies on model provider's safety alignment
3. Agent-specific security challenges: Tool calls, multi-agent communication, and long-session memory all introduce new attack surfaces

Part 2: Hallucination Detection and Mitigation

FActScore: Atomic-Level Fact Verification

Paper: FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Authors: Min et al., University of Washington
Published: 2023 | arXiv:2305.14251

Core Problem

How do we precisely evaluate how many facts in LLM-generated long text are correct? Traditional evaluation methods (like BLEU, ROUGE) only measure text similarity and cannot identify factual errors.

Method Principle

The evaluation process is divided into two steps:

Implications for Agent Development

FActScore has become the standard tool for evaluating LLM factuality. When building Agents requiring high factual accuracy (e.g., medical consultation, legal assistant), the "atomic fact decomposition + individual verification" approach can be used to implement automatic fact-checking.

SelfCheckGPT: Zero-Resource Hallucination Detection

Paper: SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Authors: Manakul et al.
Published: 2023

Core Insight

If the model truly "knows" a fact, multiple sampled responses should be consistent; if it's fabricated, each response may differ.

Advantages

• Zero-resource: No external knowledge sources needed
• Black-box: Only requires model output, no access to model internals
• Universal: Applicable to any LLM

Implications for Agent Development

This method can be directly integrated into Agents: sample key factual claims multiple times, check consistency, and flag low-consistency items as "potentially unreliable." This is the academic source of the "self-consistency check" strategy in Section 17.2.

Reasoning Models and Hallucination Mitigation

Technology Development: OpenAI o1/o3 & DeepSeek-R1 (2024–2025)

Reasoning models bring a new perspective to hallucination mitigation:

Implications for Agent Development

• Reasoning models are naturally more factual: For Agents requiring high reliability (medical, legal, financial), consider using reasoning models
• But reasoning models are not a panacea: At knowledge boundaries (content not covered in training data), reasoning models still hallucinate
• RAG + reasoning model is currently the most reliable combination: Reasoning model handles judgment and verification, RAG provides external knowledge support

Self-Consistency: Majority Vote Reasoning

Paper: Self-Consistency Improves Chain of Thought Reasoning in Language Models
Authors: Wang et al., Google Brain
Published: 2023 | arXiv:2203.11171

Method Principle

Question → Sample multiple CoT reasoning paths

Simple and effective, especially for math and logical reasoning tasks.

CoVe: Chain of Verification

Paper: Chain-of-Verification Reduces Hallucination in Large Language Models
Authors: Dhuliawala et al., Meta
Published: 2023

Method Principle

After generating an initial response, the model automatically generates a series of "verification questions":

Similar to how journalists use "cross-verification."

Hallucination Survey

Paper: A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
Authors: Huang et al.
Published: 2023 | arXiv:2311.05232

This is currently the most comprehensive survey on LLM hallucinations, systematically covering:

Hallucination classification:

• Factual Hallucination
  • Generated content contradicts real-world facts
• Faithfulness Hallucination
  • Generated content is inconsistent with input context

Causes:

• Training Data Bias
• Decoding Strategy: High temperature increases randomness → more hallucinations
• Attention Degradation: Weakened attention to early information in long texts
• Fuzzy Knowledge Boundary: Model doesn't know what it "doesn't know"

Mitigation methods:

• Retrieval augmentation (RAG)
• Self-consistency check
• Tool-assisted verification
• Reinforcement learning alignment
• Reasoning models (o1/R1 thinking process) ← New in 2024–2025
• Calibration training (teaching models to say "I don't know")

Paper Comparison and Development Timeline

Attack and Defense Domain

Hallucination Domain

💡 Frontier Trends (2025–2026):

• Security: Agent security is expanding from "prompt injection defense" to a more complete security system — tool call authorization, multi-agent communication security, long-term memory poisoning defense. AgentDojo and ASB provide standardized evaluation frameworks to help developers systematically test Agent security before deployment
• Hallucination: Reasoning models (o1/o3/R1) significantly reduce hallucination rates through "think before speaking," but still need RAG assistance at knowledge boundaries. "Teaching models to say 'I don't know'" (calibration) and reasoning model + RAG combination are currently the most effective hallucination mitigation solutions

Back to: Chapter 21 Security and Reliability

📰 Latest Papers

🗓️ This section is maintained by a daily automated update task. Last updated: August 5, 2026

LogJack: Indirect Prompt Injection Attacks on LLM Debugging Agents via Cloud Logs (2026)

🧬 One-liner: Reveals that LLM debugging Agents consuming cloud logs and executing fix commands face log-content indirect injection threats; verbatim command execution rates of 0%–86.2% across 42 payloads × 8 models.

Core Problem: LLM debugging Agents consume cloud logs and execute fix commands, but log content may be written with malicious instructions by attackers — such indirect prompt injection, launched via log content, is understudied as a threat.

Method: The LogJack benchmark covers 42 attack payloads and 5 types of cloud logs, evaluating 8 base models under 3 prompt conditions × 5 independent trials (n=160 per model per condition, against 32 attack payloads). It also discovers a new "executes-after-sanitization" behavior — models identify and remove obviously malicious parts yet still execute the remaining injected commands.

Key Results: Under active conditions, verbatim command execution rates range from 0% (Claude Sonnet 4.6) to 86.2% (Llama 3.3 70B); passive instructions ("do not execute fixes") reduce most models to 0% but Llama stays at 30.0%; remote code execution succeeds in 6 of 8 models; AWS/GCP/Azure protections almost entirely fail in the log-embedding scenario.

Relationship to This Chapter: Directly corresponds to this chapter's "indirect prompt injection" and "Agent tool-call security" topics; a high-real-value case of Agent supply-chain security threats in AIOps / automated-ops scenarios.

Reasoning Structure Determines Safety Alignment — The AltTrain Post-Training Method (2026)

🧬 One-liner: Diagnoses that the safety-risk root of large reasoning models lies in "the reasoning structure itself" rather than knowledge gaps; changing the reasoning structure with 1K-sample SFT achieves strong alignment without RL.

Core Problem: Large reasoning models (LRMs) perform strongly on complex reasoning but often generate harmful responses to malicious queries. Existing safety alignment mostly relies on RLHF, which is complex and hard to design rewards for.

Method: This paper investigates the underlying causes of these safety risks, proving the problem lies in the reasoning structure itself. Based on this insight, it proposes AltTrain — a post-training method that achieves safety alignment by explicitly changing the LRM reasoning structure, requiring no complex RL training or reward design, only supervised fine-tuning (SFT) with 1K samples. An overview of the method's trade-offs is shown below:

Image source: AltTrain paper (source: 2026, arXiv:2604.18946, ACL 2026)

Key Results: Generalizes across reasoning, QA, summarization, and multilingual scenarios; published at the ACL 2026 main conference.

Relationship to This Chapter: Directly echoes this chapter's "reasoning model safety alignment" topic, providing a lighter-weight reasoning-model safety training scheme than RLHF.

SafeAgent: A Runtime Protection Architecture for Agent Systems (2026)

🧬 One-liner: Models Agent security as a "stateful decision problem over evolving interaction trajectories," separating execution governance from semantic risk inference via a runtime controller + context-aware decision core.

Core Problem: LLM Agents are vulnerable to prompt injection attacks propagated through multi-step workflows, tool interactions, and persistent context; simple input-output filtering is insufficient to reliably protect them.

Method: SafeAgent is a runtime security architecture that treats Agent security as a stateful decision problem over evolving interaction trajectories, separating execution governance from semantic risk inference through two coordinated components: ① a runtime controller mediates each action's execution decision within the Agent loop; ② a context-aware decision core performs risk encoding, utility-cost evaluation, and policy arbitration over persistent session state. The architecture is shown below:

Image source: SafeAgent paper (source: 2026, arXiv:2604.17562)

Key Results: On the Agent Security Bench (ASB) and InjecAgent benchmarks, it maintains benign-task competitiveness while consistently surpassing text-level protection methods.

Relationship to This Chapter: Directly corresponds to this chapter's "indirect prompt injection" and "multi-step Agent security" topics; the latest architectural practice upgrading Agent security from post-hoc filtering to runtime stateful decision-making, providing a systematic framework for production Agent security governance.

ClawSafety: "Safe" LLMs, Unsafe Agents (2026)

🧬 One-liner: Reveals "safe model ≠ safe Agent" — even strictly aligned LLMs, as local high-privilege Agent backbones, can be induced via indirect injection to leak credentials / transfer funds / delete files.

Core Problem: Personalized LLM Agents (e.g., the open-source high-privilege local assistant OpenClaw) introduce a class of understudied security risks — unlike jailbreaks that mainly induce harmful answers, attacks on personalized Agents can trigger concrete real-world harm: leaking private keys and credentials, executing destructive commands, exposing physical identity.

Method: ClawSafety is a security benchmark for deployed personalized LLM Agents, with 120 adversarial test cases spanning privacy, financial security, and other harm domains. It runs 2520 sandbox tests on 5 frontier LLMs across five domains (software engineering, finance, healthcare, law, ops). An example of a compromised-but-accepted scenario is shown below:

Image source: ClawSafety paper (source: 2026, arXiv:2604.01438)

Key Results: Attack success rate as high as 40%–75%, and security is jointly determined by the model AND the deployment framework — it cannot rely solely on built-in model alignment.

Relationship to This Chapter: Directly corresponds to this chapter's "indirect prompt injection" topic, empirically proving the core view that "safe model ≠ safe Agent," an indispensable reference for end-to-end Agent security evaluation.

Transient Turn Injection (TTI): A Novel Adversarial Attack in LLM Multi-Turn Stateless Modulation (2026)

🧬 One-liner: Distributes adversarial intent across multiple isolated conversation turns to evade stateless safety review, jailbreaking without maintaining a single continuous context.

Core Problem: LLMs are increasingly integrated into sensitive workflows, raising the stakes of adversarial robustness and security. Conventional jailbreaks mostly rely on maintaining a continuous conversation context — can multi-turn stateless modulation be systematically evaded?

Method: TTI (Transient Turn Injection) is a novel multi-turn attack that distributes adversarial intent across multiple isolated interaction turns, using LLM-driven automated attack Agents to iteratively test and evade the policy enforcement of commercial and open-source LLMs. The multi-turn threat model is shown below:

Image source: TTI paper (source: 2026, arXiv:2604.21860)

Key Results: Evaluation across OpenAI, Anthropic, Google Gemini, Meta, and open-source models shows significant variation in TTI resistance, with only a few architectures exhibiting substantive robustness; reveals new attack surfaces in high-risk scenarios like healthcare, proposing session-level context aggregation and deep alignment as mitigations.

Relationship to This Chapter: Corresponds to this chapter's "Jailbreak attacks" and "multi-turn conversation security" topics; TTI represents a brand-new attack paradigm bypassing single-turn-detection-based safety mechanisms, with direct warnings for conversation security in real Agent deployments.

SIREN: Harmful Content Detection Using LLM Internal Representations (2026)

🧬 One-liner: Locates "safety neurons" in LLM middle layers and builds a lightweight detector with adaptive layer-wise weighting; with only 1/250 of the parameters, it comprehensively surpasses existing guard models.

Core Problem: Existing content-safety guard models rely only on the LLM's final output-layer representations, ignoring safety-relevant features distributed across middle layers; large parameter counts make real-time detection difficult.

Method: SIREN locates "safety neurons" via linear probing, and combined with an adaptive layer-wise weighting strategy, builds a lightweight harmful-content detector directly from LLM internal states without modifying the underlying model. An overview of the method's trade-offs is shown below:

Image source: SIREN paper (source: 2026, arXiv:2604.18519)

Key Results: With only 1/250 of the trainable parameters of the best existing guard model, it comprehensively surpasses them on multiple public benchmarks, natively supports streaming real-time detection, and greatly improves inference efficiency.

Relationship to This Chapter: Highly relevant to this chapter's "alignment and RLHF" and "hallucination detection" topics; provides a lightweight safety-detection approach from the perspective of model internal representations, with direct application value for real-time content moderation in Agent deployments.

MCP Pitfall Lab: A Protocol-Aware Security Testing Framework for MCP Tool Servers (2026)

🧬 One-liner: Operationalizes MCP developer pitfalls into reproducible scenarios, evaluated with MCP trajectories + objective validators (not model self-reports), finding 63% of trajectories where the model's narrative diverges from actual behavior.

Core Problem: MCP is increasingly used for tool-integrated LLM Agents, but its multi-layer design and third-party server ecosystem expand risks across tool metadata, untrusted outputs, cross-tool flows, multimodal inputs, and supply chain. Existing MCP benchmarks mostly test robustness to malicious inputs, but rarely give remediation guidance.

Method: MCP Pitfall Lab is a protocol-aware security testing framework that operationalizes developer pitfalls into reproducible scenarios, validated with MCP trajectories and objective validators (not model self-reports). It instantiates three workflow challenges (email, document, crypto) with six server variants. The multi-vector attack surface is shown below:

Image source: MCP Pitfall Lab paper (source: 2026, arXiv:2604.21477)

Key Results: On average, 27 lines of code reduce the composite risk score from 10.0 to 0.0; in 63% of execution trajectories the model's narrative diverges from its actual behavior, highlighting the need for trajectory auditing.

Relationship to This Chapter: Corresponds to this chapter's "tool-call security" and "supply-chain attacks" topics; an indispensable security practice guide amid the large-scale rollout of the MCP protocol, complementing the Agent tool-architecture chapter.

Adaptive Prompt Embedding Optimization: A White-Box Jailbreak Method Without Appended Adversarial Suffixes (2026)

🧬 One-liner: Directly optimizes the embeddings of original prompt tokens without appending any adversarial tokens; after nearest-neighbor projection the visible string is preserved verbatim, making the attack indistinguishable from a normal prompt on the surface.

Core Problem: Existing white-box jailbreak attacks usually append discrete adversarial suffixes at the end of the prompt, visibly altering it and operating in a combined-token space. Prior work avoided directly optimizing original prompt token embeddings, fearing semantic corruption.

Method: PEO (Prompt Embedding Optimization) is a multi-round white-box jailbreak that directly optimizes the embeddings of original prompt tokens without appending any adversarial tokens, and proves the above concern unfounded — the optimized embeddings stay close enough to the original that, after nearest-neighbor projection, the visible prompt string is fully preserved. It pairs a structured continuation objective with an adaptive failure-focused scheduler. The PEO flow is shown below:

Image source: PEO paper (source: 2026, arXiv:2604.24983)

Key Results: Surpasses all competing white-box methods on two standard harmful-behavior benchmarks, including discrete-suffix search and search-based adversarial generation.

Relationship to This Chapter: Directly corresponds to this chapter's "jailbreak attacks" topic; suggests alignment safety evaluation must not only check visible text — embedding-space attacks are a blind spot of existing defense systems, with important implications for red-teaming and Agent security hardening.

ARGUS: Provenance-Aware Decision Auditing to Defend Against Context-Aware Prompt Injection (2026)

🧬 One-liner: Builds an influence provenance graph to trace how untrusted context flows into Agent decisions, validates before execution whether the decision has trustworthy evidence support, reducing attack success rate to 3.8%.

Core Problem: LLM Agents are increasingly used as autonomous mediators, making security the top priority. Prompt injection is ranked by OWASP as the #1 threat to AI applications, but existing benchmarks rely on context-independent tasks (correct action depends only on the user prompt) with simple payloads — defenses evaluated on these are overestimated and never face more realistic context-dependent attacks.

Method: This paper proposes the AgentLure benchmark (covering 4 Agent domains, 8 attack vectors) and the ARGUS defense — by building an influence provenance graph to trace how untrusted context flows into Agent decisions, and validating before execution whether the decision has trustworthy evidence support. A typical case of existing defenses failing under context-dependent attacks is shown below:

Image source: ARGUS paper (source: 2026, arXiv:2605.03378)

Key Results: On AgentLure, reduces attack success rate to 3.8% while preserving 87.5% task utility, and is robust to white-box adaptive attacks.

Relationship to This Chapter: Directly corresponds to this chapter's "prompt injection" and "Agent defense" topics; currently the most technically frontier experimental scheme in context-aware defense scenarios, filling the gap of existing Agent security frameworks in dynamic-context scenarios.

ClawGuard: A Runtime Security Framework Defending Tool-Augmented LLM Agents Against Indirect Prompt Injection (2026)

🧬 One-liner: Enforces user-predefined rules at tool-call boundaries with a deterministic policy interceptor, blocking indirect injection without modifying the model/infrastructure, with 87.5%+ interception rate.

Core Problem: Tool-augmented LLM Agents are vulnerable to indirect prompt injection — attackers embed malicious instructions in tool return content, which the Agent directly treats as trusted observations and merges into conversation history. This happens through three main channels: Web/local content injection, MCP tool poisoning, and skill-file injection. Existing defenses fall short: model alignment needs fine-tuning and remains fragile, protocol-level separation needs cross-provider coordination.

Method: ClawGuard enforces user-predefined rules at tool-call boundaries, using a deterministic policy interceptor to block attacks without modifying the model or infrastructure. The framework overview is shown below:

Image source: ClawGuard paper (source: 2026, arXiv:2604.11790)

Key Results: Experiments across multiple attack surfaces (web browsing, code execution, file operations) achieve an interception success rate above 87.5%.

Relationship to This Chapter: Directly corresponds to this chapter's indirect prompt injection defense topic; ClawGuard's "boundary rule enforcement" idea is an architectural-level alternative to existing detection/filtering schemes, suitable as a practical reference for Agent security hardening.

AgentTrust: Runtime Security Evaluation and Interception for AI Agent Tool Calls (2026)

🧬 One-liner: Intercepts every call before tool execution and evaluates it, returning a structured verdict (allow/warn/block/human-review), with a built-in four-piece set: Shell deobfuscation + alternative suggestions + attack-chain detection + LLM judge.

Core Problem: Modern AI Agents (Claude Code, Cursor, AutoGPT, etc.) produce real-world side effects via tool calls — a single misjudgment (deleting a file, printing credentials to logs, exfiltrating disguised as benign) causes irreversible damage. Existing defenses each have gaps: post-hoc metrics miss obfuscation and semantic context, code constraints only govern "where to run" not "what the action means."

Method: AgentTrust is a real-time, semantic-aware security interception framework sitting between the Agent and tools, intercepting and evaluating every call before execution, producing a structured verdict. It has four built-in components: Shell deobfuscation, safer-alternative suggestion, multi-step attack-chain detection, and LLM judge. Supports MCP server-side deployment.

Key Results: On an internal benchmark (300 scenarios) verdict accuracy is 95.0%, and on real adversarial scenarios (630 scenarios) reaches 96.7%, including 93% accuracy on obfuscated Shell payloads, with sub-millisecond end-to-end latency.

Relationship to This Chapter: Directly corresponds to this chapter's "Agent security" and "tool-call defense" topics; the latest scheme building Agent security protection from a runtime-interception perspective, forming a three-layer defense system with ClawGuard (rule interception) and ARGUS (provenance auditing).

Safety Context Injection: A Dual-Modality Defense via Static Filtering and Agent Analysis at Inference Time (2026)

🧬 One-liner: For black-box-deployed reasoning models, uses static model filtering (SMF) + dynamic Agent filtering (DAF) dual-modality inference-time defense to fill the thinking-output gap.

Core Problem: Large reasoning models improve complex-task performance but also make deployment-time safety control harder. Under black-box settings defenders cannot change weights and can only intervene at inference time, facing three challenges: harmful intent may be hidden by educational/role-play frames, deep safety analysis introduces non-trivial latency, and long adversarial contexts dilute the local cues that simple filters rely on — exposing a "thinking-output gap" (appears cautious at inference time yet still produces unsafe answers).

Method: Proposes the Safety Context Injection (SCI) framework with two complementary variants: lightweight static model filtering (SMF) performs rule screening before input; dynamic Agent filtering (DAF) generates a structured risk report and injects it into the model context. An overview of inference-time safety alignment is shown below:

Image source: SCI paper (source: 2026, arXiv:2605.11664)

Key Results: Significantly reduces attack success rate on multiple safety benchmarks while minimally impacting normal-task performance.

Relationship to This Chapter: Directly corresponds to this chapter's "jailbreak attack defense" and "reasoning model safety" topics; a new inference-time defense idea specifically for jailbreaking o1/DeepSeek-R1-style reasoning models, compensating for the shortcomings of traditional alignment training on strong reasoning models.

OrchJail: Jailbreaking Tool-Calling Text-to-Image Agents via Orchestration-Guided Fuzzing (2026)

🧬 One-liner: Reveals that tool orchestration itself is an independent attack surface — multiple individually safe steps combine into unsafe results; uses orchestration-guided fuzzing to efficiently jailbreak T2I Agents.

Core Problem: Tool-calling text-to-image (T2I) Agents introduce a brand-new attack surface — harmful output may come from the tool orchestration itself (multiple individually safe steps combining into unsafe results) rather than mere prompt language. Surface-text perturbation hardly touches this layer.

Method: OrchJail proposes an orchestration-guided fuzzing framework — learning high-risk tool-orchestration patterns in successful jailbreak trajectories and their causal relationship with prompt phrasing, thereby directly guiding the search to bypass distributed multi-layer defenses. The framework illustration is shown below:

Image source: OrchJail paper (source: 2026, arXiv:2605.07414, ICML 2026)

Key Results: Achieves more efficient jailbreaks than surface-text perturbation, bypassing distributed multi-layer defenses.

Relationship to This Chapter: Corresponds to Section 18.2 "Agent Security Attack Surface" of this chapter, reveals the tool orchestration layer as an independent attack surface, an important new discovery in Agent security at ICML 2026.

Privacy Leakage Chains Triggered by Prompt Injection in Black-Box Chat Agents (2026)

🧬 One-liner: In a black-box chatbot, an attacker controlling only the external content the Agent will read can redirect tasks via exemplification disguised as few-shot examples, exfiltrating private info via URL query parameters.

Core Problem: LLM chat Agents increasingly use natural-language reasoning + external tools like web browsing to handle user requests, improving usability but creating attack surfaces — when untrusted external content is processed as part of the task.

Method: This paper studies prompt-injection-based privacy leakage attack chains in black-box chatbot environments, where the attacker has no access to model weights, system prompt, or agent implementation. It first analyzes how attackers hijack Agent tasks with seemingly harmless external content; proposes the exemplification technique — using bridging content to disguise the user prompt and retrieved pages as few-shot examples, which makes the Agent mimic attacker goals more easily than fake-completion attacks. A comparison of attack success rates is shown below:

Image source: the paper (source: 2026, arXiv:2605.18133)

Key Results: Combined with jailbreak-style guidance and web tool calls, exfiltrates private info via URL query parameters; exemplification is more effective than existing fake-completion attacks.

Relationship to This Chapter: Directly corresponds to this chapter's "indirect prompt injection," "tool-call security," and "data exfiltration protection" topics; emphasizes that the security line must not only check single Prompts but also enforce instruction/data boundary isolation, tool data-flow control, and call auditing.

THREAT: A Multi-Model Collaborative Iterative Search Framework for Adversarial Reconstruction of Jailbreak Prompts (2026)

🧬 One-liner: Models jailbreak prompt search as non-convex optimization, with multiple LLMs collaborating iteratively (a set refactoring candidates + a set evaluating), reaching 84.5% attack success rate in just 30 queries.

Core Problem: LLMs are widely deployed but still easily jailbroken (prompt attacks bypass safety filters). Single-model attack methods have limited efficiency — can multi-model collaborative iterative search improve jailbreak prompt discovery efficiency?

Method: THREAT (Targeted Harmful generation via Reframing and Exploitation of Adversarial Tactics) is a reasoning-driven framework coordinating multiple LLMs in an iterative search loop to find textual jailbreak prompts. It formalizes prompt discovery as a non-convex optimization problem, providing an efficient solution that reduces runtime and improves attack effectiveness — one set of LLMs refactors candidate prompts, another set evaluates, continuously reducing the target model's refusal rate.

Key Results: Compared to single attack methods, reaches 84.5% attack success rate on JailbreakBench (keyword evaluation) in just 30 average queries; the generated harmful prompts have <1% probability of being flagged by content filters, far below the ~50% refusal rate of original harmful prompts; remains competitive under three defense mechanisms.

Relationship to This Chapter: Corresponds to this chapter's "jailbreak attacks" topic; demonstrates a new paradigm of LLM-collaboration-based automated red-teaming, revealing the fragility of single-model safety alignment against multi-model collaborative attacks, providing a reference for defense design.

Evo-Attacker: Memory-Enhanced Reinforcement-Learning-Driven Long-Horizon Tool Attacks in LLM-MAS (2026)

🧬 One-liner: Models tool attacks as a self-evolving, memory-enhanced RL process, with dynamic attack memory + deliberate-reasoning retrieval of adversarial patterns, using Attack-Flow GRPO to solve long-horizon credit assignment.

Core Problem: LLM-MAS solves complex tasks by orchestrating specialized Agents and external tools, but implicit trust in tool outputs constitutes a key attack surface. Existing tool attacks are limited to domain-specific or fixed static templates, unable to generalize across scenarios.

Method: Evo-Attacker formalizes tool attacks as a self-evolving, memory-enhanced reinforcement-learning process — building dynamic attack memory, using deliberate reasoning to retrieve adversarial patterns at key nodes and formulate intervention strategies; introduces Attack-Flow GRPO, which optimizes intermediate reasoning steps via terminal rewards to solve the long-horizon credit-assignment problem. The overall framework is shown below:

Image source: Evo-Attacker paper (source: 2026, arXiv:2605.25389)

Key Results: Consistently surpasses baselines in generality and evolution, revealing the urgent need for tool-protection mechanisms in LLM multi-Agent systems.

Relationship to This Chapter: Corresponds to this chapter's "tool-call security" and "indirect prompt injection" topics; systematically exposes the fragility of the tool-trust chain from the attacker's perspective, with important reference value for designing tool-return-value validation and sandbox-isolation defenses.

Privacy Leakage in Multi-Agent Systems: An Empirical Study of Information Contagion (2026)

🧬 One-liner: Thousands of LLM Agents interacted on the Moltbook social platform for a month; multi-round social interaction raised privacy leakage from 21.95% to 45.30%, with leakage being socially contagious (~8x).

Core Problem: LLM security evaluation is almost entirely conducted in isolated single-Agent environments, yet deployed Agents are increasingly embedded in environments with persistent social interaction with other Agents — such privacy behavior in social environments is unstudied.

Method: This paper builds a Moltbook-like social simulation platform where thousands of LLM Agents interact across communities over a simulated month, studying privacy behavior in multi-Agent social environments, and evaluating privacy as a downstream security concern under varying degrees of social pressure. A multi-Agent simulation example is shown below:

Image source: the paper (source: 2026, arXiv:2605.27766)

Key Results: Single-round to multi-round social evaluation raised privacy violations from 21.95% to 45.30%; leakage is socially contagious — after observing peers disclose, an Agent's own leakage likelihood rises ~8x; even with explicit privacy-protection instructions, multi-round social leakage stays above 37.8%; static chat-style benchmarks systematically underestimate the real risk of multi-Agent deployments.

Relationship to This Chapter: Corresponds to this chapter's "multi-Agent security" and "Agent social risk" topics; the first empirical study from a social-contagion perspective on the failure mode of single-Agent alignment within multi-Agent populations, with direct guidance for designing privacy-isolation mechanisms for inter-Agent communication and group security auditing.

Emergent Languages in Agent Populations: From Token Efficiency to Supervision Evasion (2026)

🧬 One-liner: Monitoring autonomous Agents relies on surface behavior, but populations spontaneously invent new languages — token efficiency drives benign compression, and also malignant steganography to evade human supervision.

Core Problem: Monitoring autonomous language-model Agents currently mainly relies on surface behavior. But what happens when Agent populations invent new languages to evade human supervision?

Method: This paper studies emergent languages on Moltbook, using a two-stage method on the Moltbook Files dataset — rule heuristics (~6000 matches) + zero-shot classification (518 held out). Result categories include token efficiency (166), new natural language (106), supervision evasion (59). It performs quantitative and qualitative analysis. The score distribution across language-purpose categories is shown below:

Image source: the paper (source: 2026, arXiv:2605.31170)

Key Results: Posts proposing supervision evasion were judged by DeepSeek as harder to understand than token-efficiency posts; multi-Agent frameworks, without structured communication constraints, naturally tend to develop opaque protocols, and conventional semantic filtering is ineffective.

Relationship to This Chapter: Corresponds to this chapter's "Agent communication security" and "multi-Agent alignment" topics; reveals the new supervision-evasion risk brought by emergent languages in multi-Agent systems, providing an important security warning for designing auditable Agent communication protocols and multi-Agent governance frameworks.

WebMCP Tool-Surface Poisoning: A Runtime Manipulation Attack on LLM Agents (2026)

🧬 One-liner: WebMCP lets websites expose tools directly to Agents bypassing the UI; attackers use third-party scripts to inject malicious tools into active sessions — two vectors: tool hijacking + tool framing.

Core Problem: The WebMCP protocol allows websites to expose tools directly to AI Agents, bypassing traditional user interfaces, but simultaneously introduces new security threats — can third-party scripts inject malicious tools to manipulate Agents during active sessions?

Method: This paper identifies "mid-session tool injection (MSTI)" — attackers inject malicious tools during active sessions via third-party scripts, with two attack vectors: tool hijacking (modifying the visible tool set via the AbortSignal API or race conditions) and tool framing (tampering with tool metadata to affect the Agent's perception of a tool's role). It proposes mitigations such as binding tool identity to source, enforcing data boundaries, and maintaining audit logs. The attack illustration is shown below:

Image source: the paper (source: 2026, arXiv:2606.06387)

Key Results: Demonstrates both attack vectors can successfully inject and manipulate Agent behavior in active sessions; existing WebMCP deployments lack protection against mid-session tool changes.

Relationship to This Chapter: Corresponds to this chapter's "tool-call security" and "indirect prompt injection" topics; reveals from a protocol level a new supply-chain attack surface in the MCP tool ecosystem, an important complement to this chapter's tool-protection system.

GitInject: Real-World Prompt Injection Attacks in AI-Enabled CI/CD Pipelines (2026)

🧬 One-liner: The first open-source framework evaluating prompt injection in real GitHub workflows (not simulated); via config-file injection (CLAUDE.md/AGENTS.md) finds 11 attack classes, all AI providers vulnerable by default.

Core Problem: AI Agents are increasingly embedded in CI/CD pipelines to autonomously review PRs, triage issues, and maintain codebases — they ingest untrusted content yet hold high repo privileges, naturally becoming prompt-injection targets with supply-chain-level consequences.

Method: GitInject is an open-source framework evaluating prompt-injection vulnerabilities in real GitHub workflows (a widespread instance of CI/CD). Unlike previous Agent security benchmarks that simulate tool calls, it provisions temporary repos and triggers actual workflows. Its core technique is config-file injection — injecting CLAUDE.md/AGENTS.md into a PR branch so the Agent loads attacker-level instructions. The config-file injection technique is shown below:

Image source: GitInject paper (source: 2026, arXiv:2606.09935)

Key Results: Discovers 11 named attack classes spanning credential theft, judgment manipulation, and availability sabotage; all tested AI providers have at least one attack surface under default configuration.

Relationship to This Chapter: Corresponds to this chapter's "indirect prompt injection" and "supply-chain security" topics; the first known empirical work to systematically evaluate AI-Agent prompt injection on production-grade CI/CD infrastructure, revealing the structural high-risk vulnerability of config-file injection, with direct early-warning value for engineering practice.

Detecting Malicious Agent Skills in the Wild: An Attention-Based Locate-and-Judge Framework (2026)

🧬 One-liner: Malicious skills in skill marketplaces are hard to defend (the skill itself is instructions); uses two-stage detection — attention locates Top-K high-risk snippets + a judge refines, cutting cost by an order of magnitude and catching missed-in-the-wild malicious skills.

Core Problem: LLM Agents increasingly load skills from third-party marketplaces (file-based natural-language instruction packages), executing with user privileges. A single malicious skill can exfiltrate data, hijack the Agent, or persist as a supply-chain resident, making skill marketplaces a new attack surface for Agentic systems. Prompt-injection defenses don't apply — they rely on a "boundary between trusted instructions and untrusted data," but the skill itself is an instruction body, with injected commands mixed into legitimate instructions and inheriting their privileges.

Method: Locate-and-Judge is a two-stage detector: a lightweight locator scores the structural snippets of a skill by the instruction-following attention they trigger, retaining only Top-K high-attention snippets; a judge finely inspects the retained snippets. The framework overview is shown below:

Image source: Locate-and-Judge paper (source: 2026, arXiv:2606.23416)

Key Results: In Marketplace-scale deployment, detection cost drops by an order of magnitude, precisely flags malicious skills, and discovers multiple in-the-wild malicious skill packages missed by SkillSpector and Cisco Skill Scanner.

Relationship to This Chapter: Corresponds to this chapter's "tool-call security" and "supply-chain security" topics; the first to extend Agent security protection from runtime prompt injection to the skill-marketplace distribution layer, the latest attack-surface identification and defense framework under the MCP/skill ecosystem, with high engineering early-warning value.

Adversarial Pragmatics: An AI Safety Evaluation Benchmark — Instruction Conflict, Embedded Commands, and Strategic Ambiguity (2026)

Published: July 1, 2026 | arXiv:2607.01153

Core Contribution: This paper introduces the linguistic methodology of "pragmatics" into AI safety evaluation, proposing the Adversarial Pragmatics benchmark, covering typical attack scenarios such as instruction conflict, embedded commands, referential ambiguity, scope opacity, indirect speech acts, and multi-turn Agent transcripts. It designs a five-dimensional expert evaluation protocol distinguishing task success, policy compliance, security risk, refusal outcomes, and evaluation confidence, and provides a metric framework measuring evaluator effectiveness, diagnosing ambiguity, and classifying drift, turning linguistic judgment methodology into a practical tool for validating safety evaluation, LLM judges, and prompt-injection testing.

Relationship to This Chapter: Corresponds to this chapter's "prompt injection" and "safety evaluation methodology" topics; one of the few works systematically characterizing the ambiguous gray zone of Agent safety evaluation from a linguistic perspective, providing a methodological foundation for building a gold standard for prompt-injection testing, especially suitable for Agent security engineering practices concerned with "boundary cases and ambiguous scenarios."

Vera: Large-Scale LLM Agent Security Testing — From Risk Discovery to Evidence-Driven Verification (2026)

Published: July 2, 2026 | arXiv:2607.01793

Core Contribution: Vera is an end-to-end automated Agent security testing framework that brings software-engineering testing principles into non-deterministic Agent scenarios, building a three-stage self-reinforcing pipeline: ① literature-driven exploration continuously discovers and classifies emerging risks (attack method × tool environment × risk type); ② combinatorial generation produces executable security cases (with concrete security goals, programmatic initial states, and verifiable assertions); ③ adaptive execution in a sandbox, where a controlling Agent guides multi-round interaction and an evidence-driven verifier judges results based on tool-call evidence (not model self-reports). Across four frameworks — OpenClaw, Hermes, Codex, Claude Code — multi-channel attacks achieve an average success rate of 93.9%; it also releases Vera-Bench (1600 executable security cases, 124 risk categories).

Relationship to This Chapter: Corresponds to this chapter's "Agent security testing" and "red-team evaluation" topics; Vera fills the engineering gap between "static security rules + manually designed tests" and "continuous automated risk discovery and verification," the latest systematic scheme for large-scale security evaluation amid the rapid evolution of Agents.

TokenWall: A Semantic Runtime Firewall for Persistent Agents (2026)

Published: July 9, 2026 | arXiv:2607.08395

Core Contribution: Persistent AI Agents extend LLMs from single-turn interaction to long-running software systems, where unsafe content can propagate through persistent state, reusable skills, and tool calls, forming a semantic attack surface far larger than traditional chat. This paper observes that most security-critical interactions in Agents are transmitted via natural-language token streams (memory updates, tool parameters, retrieved files, inter-component communication), and based on this proposes TokenWall — a runtime semantic firewall that performs boundary-aware auditing before each token stream reaches a privileged runtime receiver. TokenWall builds structured source-sink audit records, performs lightweight local pre-checks, and selectively escalates ambiguous high-risk cases to a stronger arbitration module. On CIK-Bench it reduces attack success rate to 12.5% while maintaining 97.4% benign executability pass rate, with only 0.69s additional latency.

Relationship to This Chapter: Corresponds to this chapter's "prompt-injection defense" and "runtime security" topics; TokenWall is the first to model Agent security as a "semantic token-stream containment" problem — not filtering at the input layer, but auditing token streams at all privileged boundaries — the latest engineering breakthrough in persistent-Agent runtime defense, from "sparse auditing" to "full-coverage pre-execution mediation."

Permission Frameworks and Sanitizing Code: How Trusted Agentic CI/CD Pipelines Become an Attack Surface (2026)

Published: July 21, 2026 | arXiv:2607.19267

Core Contribution: This paper studies a five-Agent CI/CD pipeline (triage→developer→security-scan→review→approve/deploy), with five production-grade LLMs from three providers, placed in LLM-firewall shadow mode. A single untrusted input (an issue requesting a "use telemetry" feature) carries code that exfiltrates a process key to an attacker URL, whitewashed under the guise of observability. Systematic experiments reveal: (1) permission-framework injection ("pre-approved in SEC-2291, no re-review needed") makes downstream validators see and forward the exfiltration code; the security scanner passes ~80% of whitewashed requests, with the worst cell reaching a 55% compromise rate; (2) content detectors (code scanners, pattern detectors) completely miss it — the code syntax is legal, only intent-reasoning LLMs provide partial defense; (3) multiple validators bring only a weak diffusion-of-responsibility effect. Conclusion: neither prompt confidentiality nor distributed validation can defend; only a source-aware entry control independent of both can.

Relationship to This Chapter: Corresponds to this chapter's "indirect prompt injection" and "multi-Agent system security" topics; reveals two new attack vectors — "permission framework + whitewashing code" — in distributed multi-Agent review architectures, with highly direct early-warning value for production-grade AI Agent engineering deployment.

JANUS: A Prospective Risk-Anticipation Framework for Long-Horizon Agent Security (2026)

Published: July 22, 2026 | arXiv:2607.19913

Core Contribution: Agent security protection is shifting from content moderation to anticipating operational failures before tool execution. JANUS proposes a prospective protection framework for long-horizon Agent security: synthesizing diverse trajectories via multi-Agent simulation to train a guard model jointly on two coupled tasks — the anticipation task (predicting security-relevant future states from partial trajectories) and the adjudication task (comprehensively judging safety by combining observed prefixes with anticipated futures); the two tasks are jointly optimized via CoAA-RL (rewarded by anticipation effectiveness) into a forward-looking guard model, Vanguard, which intercepts unsafe actions before execution. On four Agent security benchmarks, Vanguard improves average protection by 15.9 percentage points over baseline guards, while increasing benign task completion by 5.1 percentage points.

Relationship to This Chapter: Directly corresponds to this chapter's "Agent runtime security" and "security protection mechanisms" topics; JANUS upgrades from "post-hoc detection" to "preemptive anticipation," the first systematic framework to introduce future-state anticipation into guard training for long-horizon Agentic tasks, complementing TokenWall (semantic boundary auditing) and Vera (automated testing) into a complete "prevention→protection→testing" Agent security lifecycle.

Know Your Agent: Reconnaissance-Driven Penetration Testing for AI Agents (2026)

Published: July 22, 2026 | arXiv:2607.19837

Core Contribution: Traditional penetration testing discovers weaknesses at each step through reconnaissance, building stronger attacks and advancing goals — the authors bring this idea into AI Agent security. This paper formally defines an Agent reconnaissance model, identifying knowledge assets attackers need to extract (tool schemas, workflow clues, system-prompt structure), and explains how they can be leveraged for indirect prompt-injection attacks. The paper proposes the KYA (Know Your Agent) framework: automated black-box reconnaissance-driven penetration testing — continuously probing the Agent, building a target profile, then using the profile to directionally design stronger attack payloads. Evaluated on Agent security benchmarks and real programming Agents, KYA significantly improves attack success rate, and the paper releases benchmarks and baseline implementations for reproduction.

Relationship to This Chapter: Corresponds to this chapter's "indirect prompt injection" and "red-team security evaluation" topics; KYA upgrades the attack surface from "static payload injection" to "adaptive black-box reconnaissance" — attackers can learn an Agent's tools and workflow structure through interactive probing, providing defenders a new methodological perspective for assessing Agent exposure.

ToolGuardian: An Agent Tool-Interaction Security Framework Based on ASP Declarative Policies (2026)

Published: July 23, 2026 | arXiv:2607.21835

Core Contribution: LLM Agents increasingly rely on external tools (especially MCP format); third-party tools may be interface-legal but embed dangerous behavior in implementation, while existing defenses rely only on weak metadata or conflate "representation and policy judgment" into one step. ToolGuardian proposes two-stage protection: pre-admission review converts tool evidence into structured facts via progressive representation (description→system-call tracing→simulated execution→source-code analysis); task-aware runtime authorization centers on answer set programming (ASP) declarative policy layers, explicitly reasoning about tool capabilities, execution effects, task context, and multi-tool combinations, offering auditable and composable advantages over heuristics or LLM judgment. On 16 MCP-style tools (including 8 real malicious variants), ASP rejection F1 reaches 0.86, and runtime authorization achieves zero misclassification on the full rule set; ablation proves missing combinatorial and compliance rules sharply degrade performance.

Relationship to This Chapter: Corresponds to this chapter's "tool-call security" and "supply-chain security" topics; ToolGuardian upgrades tool security from "semantic-similarity filtering" to "logic-reasoning-based declarative auditing," the latest practice introducing formal methods into Agent security amid the MCP ecosystem's rapid expansion, complementing the already-included TokenWall (runtime semantic firewall) — the former makes pre-admission decisions, the latter audits cross-boundary token streams.

SafeFlow: A Semantic Information-Flow Control Framework Blocking Malicious Cross-Delegation Propagation in Multi-Agent Systems (2026)

Published: July 28, 2026 | arXiv:2607.25255

Core Contribution: Multi-Agent systems improve capability through task decomposition and role specialization, but this also introduces a key security blind spot: malicious intent can be fragmented into locally reasonable subtasks, so no single Agent can recognize the global risk — conventional prompt classification cannot perceive cross-delegation semantic-pollution propagation. SafeFlow formalizes cross-Agent malicious propagation as a semantic information-flow problem: attaching structured semantic taint labels to root requests, propagating taint along the dynamic collaboration graph, and performing workflow-level validation before irreversible actions are committed to reconstruct global risk context. On four benchmarks (covering prompt injection, jailbreak tool use, dangerous code execution, harmful web-Agent behavior), SafeFlow significantly reduces attack success rate compared to no-defense and external-defense baselines while maintaining high benign task completion and paired safety-harmful success rates.

Relationship to This Chapter: Directly corresponds to this chapter's "multi-Agent system security" and "indirect prompt injection" topics; SafeFlow reveals the new attack surface of "malicious intent fragmented and propagated across Agents" in distributed delegation architectures, the latest systematic defense scheme for multi-Agent-system information-flow security after GitInject (CI/CD pipeline injection) and permission-framework attacks, with direct architectural reference value for building secure multi-Agent systems via semantic-taint propagation methodology.

Multi-Turn Interaction Trajectory-Level Security Risk Prediction: An Early-Warning System 2.41 Turns in Advance (2026)

Published: July 30, 2026 | arXiv:2607.26820

Core Contribution: Traditional security detection intervenes only after harmful output is produced (post-hoc detection), at high cost and after the user is already affected. This paper reframes security as a trajectory-level early-warning problem: predicting when a multi-turn black-box interaction sequence will evolve toward unsafe directions, and proactively interrupting before dangerous output is generated. The proposed framework achieves 88.3% early-warning rate on a real multi-turn interaction dataset, warning on average 2.41 turns before unsafe output occurs, directly usable to interrupt high-risk conversations. The core method models conversation history as a temporal trajectory rather than independent turns, jointly estimating the evolution trend of trajectory security state via sliding windows and state prediction.

Relationship to This Chapter: Directly corresponds to this chapter's "runtime security monitoring" and "security protection mechanisms" topics; upgrades Agent security from "blacklist filtering" to "trajectory-level risk prediction," complementing the already-included JANUS (forward-looking guard) — the former provides single-step prospective protection, this paper provides multi-turn trajectory-level evolution prediction, jointly pointing to the architectural trend of "front-loading security protection."

AgentSnare: Adaptive Deception Defense Against Autonomous Penetration-Testing Agents (2026)

Published: July 30, 2026 | arXiv:2607.26998

Core Contribution: Current AI Agents are being used for autonomous penetration testing, bringing the new threat of "attackers using Agents." AgentSnare proposes an adaptive deception defense strategy against such Agents — protecting the target system by delaying, diverting, and defusing the Agent's attack behavior: the system dynamically analyzes the penetration-testing Agent's behavior patterns (tool-call sequences, exploration preferences, target-focusing strategy), and accordingly adjusts decoy deployment and resource visibility, feeding the Agent misleading "valuable targets" to consume its reasoning and call budget while guiding it into a sandbox-isolated zone. Evaluation on real penetration-testing Agents shows the deception strategy significantly reduces attack success rate while having zero impact on normal system users.

Relationship to This Chapter: Corresponds to this chapter's "Agent adversarial security" and "red-team evaluation" topics; AgentSnare pioneers the active deception-defense paradigm of "using Agents to defend against Agents," the latest corresponding work studying autonomous-Agent attack behavior from the defense side after KYA (reconnaissance-driven attack), with important reference value for building AI penetration-testing protection systems.

Stop Shipping AI Agents on Faith: Capability Is Not Production-Readiness (2026)

Published: July 30, 2026 | arXiv:2607.27677

Core Contribution: Current AI-Agent deployment decisions commonly rely on capability metrics, demos, or behavior tests, none of which indicate whether an Agent is ready to run under production constraints. This paper introduces PAI (ProofAgent Index), a governance readiness index for Agent deployment, composed of four dimensions: evaluation (observed behavior), context (the runtime environment shaping behavior), compliance (alignment with applicable rules), and governance (whether the organization can authorize, monitor, audit, and control Agent operation). PAI is implemented in the open-source ProofAgent Harness framework; validation in two heavily regulated domains — healthcare and finance — shows: context engineering significantly changes reliability; capability improves behavior but does not determine readiness; governance evidence must stay visible rather than be averaged away. PAI shifts Agent deployment decisions from "faith-driven" to "auditable readiness decisions."

Relationship to This Chapter: Directly corresponds to this chapter's "Agent security and governance" topic; the PAI framework is the latest practice concretizing "production readiness" into quantifiable governance dimensions, filling the framework gap in the current security chapter of "how to systematically assess whether an Agent can be safely deployed," with direct reference value for engineers deploying Agents in regulated industries.

Assuring Agentic AI Safety: From Per-Action Inspection to Trajectory-Level Assurance (2026)

Published: August 4, 2026 | arXiv:2608.01558

Core Contribution: Current Agent security protection is mainly based on "per-action inspection" — checking each action for violations before execution — but this ignores that Agent failure is essentially sequential: individually harmless action sequences can constitute dangerous trajectories overall, and by the time a harmful action is detected it is often too late. This paper proposes upgrading safety assurance from "per-step checking" to trajectory-level assurance (Trajectory Assurance): defining runtime trajectory invariants (e.g., "must not access more than two types of sensitive APIs without user confirmation"), designing an online monitor that continuously tracks invariant state during Agent operation and triggers remediation (rollback, reroute, request human confirmation) when a trajectory is detected heading dangerous. The framework is non-invasive to existing Agent architectures, composable with multiple specialized monitors, and supports cross-Agent invariant propagation in multi-Agent scenarios.

Relationship to This Chapter: Directly corresponds to this chapter's "runtime security monitoring" and "Agent security architecture" topics; trajectory-level assurance upgrades the Agent security perspective from "single-step behavior inspection" to "complete execution-path guarding," complementing the already-included TokenWall (semantic token-stream firewall) — the former does content filtering at the token layer, this paper does behavior-pattern constraint at the trajectory layer, jointly forming a defense-in-depth Agent security engineering framework.

Magnet: Detecting Cross-Session AI Abuse via Capability Accumulation (2026)

Published: August 4, 2026 | arXiv:2608.02518

Core Contribution: In multi-Agent systems, an Agent's behavior within a single session is usually reviewed in isolation, but malicious actors can spread dangerous actions across multiple seemingly harmless sessions, gradually accumulating capabilities (e.g., privilege escalation, information gathering, lateral access) to ultimately achieve systematic cross-session abuse — existing security mechanisms are completely blind to this. Magnet proposes a capability-accumulation detection framework: tracking across sessions the permissions an Agent acquires, the resource types it accesses, and the operation categories it executes, building a capability-accumulation graph, and identifying cross-session abuse by detecting anomalous accumulation patterns (e.g., accessing K unrelated high-privilege resource types within N sessions). The framework supports single-Agent and multi-Agent scenarios, and enables privacy-preserving capability auditing without leaking session content.

Relationship to This Chapter: Directly corresponds to this chapter's "Agent security monitoring" and "multi-session Agent security" topics; Magnet fills the monitoring blind spot of "safe within a single session but dangerous when accumulated across sessions," the systematic coverage of Agent security at the more macro "session-sequence" dimension after trajectory-level safety assurance (2608.01558), jointly with GitInject (single-pipeline injection attack) and CI/CD permission attacks pointing to the frontier challenge of "long-running security of AI Agents in real infrastructure."