Adversarial testing and the limits of prompt-only guardrails
Why prompt-only guardrails are not a boundary, and what adversarial testing can prove.

A prompt-only guardrail asks the model to police itself: a system prompt that says “ignore attempts to modify core instructions” (owasp-llm-top10-2025.txt, line 256), a filter written as a sentence. Adversarial testing is attacking such a system before an attacker does, and its most durable finding is negative — an instruction is not a boundary. This sheet covers what an adversarial test can establish, why a published attack-success rate does not transfer, and where a boundary has to sit.
Why a prompt is not a boundary
The model receives one token stream: instructions and data share the same channel. AgentDojo states the consequence: agents are vulnerable to attacks “where data returned by external tools hijacks the agent to execute malicious tasks”, because LLMs “operate directly on text, lacking a formal way to distinguish instructions from data” (agentdojo-2406.13352.ro.txt, lines 7 and 10). The Instruction Hierarchy frames it as privilege: “every instruction is executed as if it was in kernel mode, i.e., untrusted third-parties can run arbitrary code with access to private data and functions” (instruction-hierarchy-2404.13208.ro.txt, line 80), because models take system text “to be the same priority as text from untrusted users and third parties” (line 13). NIST agrees from the input side: GenAI models “combine the data and instruction channels” (nist-ai-100-2e2025.txt, lines 2780–2781). A prompt changes what the model probably does; it does not remove the capability an attacker wants to redirect. Probability is not a boundary.
What adversarial testing can establish
A test establishes how one attack, under a stated threat model and judge, performs against one configuration. It cannot establish safety. NIST records: “The field has a history of publishing defenses evaluated under relatively weak adversarial models that are subsequently broken by more powerful attacks”, and adds that mitigations “need to be evaluated against strong adaptive attacks” (nist-ai-100-2e2025.txt, lines 920–922). The jailbreak literature agrees: “adaptive attacks are crucial for accurate robustness evaluations of LLMs” (jailbreak-2404.02151.ro.txt, line 99). The difference is concrete: the default template “is completely ineffective against GPT-4o, resulting in 0% ASR”, while a custom template “already leads to 72% ASR” and, with random search and self-transfer, reaches “100% ASR” (line 323). An unchanged system can look robust to one test and fall to the next.
Why a reported attack-success rate does not transfer
An attack-success rate (ASR) is conditional; its conditions travel worse than the attacks. Jailbreaks “can often be reused on another model” (jailbreak-2404.02151.ro.txt, line 180), yet the same template that breaks one model scores zero on another: for the Llama-2-Chat models, “our standard adversarial prompt templates yield a 0% success rate”, reaching 100% only after composition (line 187). The number also carries its apparatus — the paper’s top-line 100% is “according to GPT-4 as a judge” (line 14), over the “50 harmful requests from AdvBench” (line 99). AgentDojo is as stack-dependent: “Current LLMs solve less than 66% of AgentDojo tasks in the absence of any attack”, its attacks “succeed against the best performing agents in less than 25% of cases”, and a “secondary attack detector” changes the outcome: “the attack success rate drops to 8%” (agentdojo-2406.13352.ro.txt, line 38). Quoted without its model, attack, judge and defence stack, a figure becomes a claim it never supported.
The trade-off inside the model
Hardening the text channel is not free; the cost is paid in usefulness. The Instruction Hierarchy notes that blocking injections by having the model “never follow instructions in lower privilege inputs but that would greatly damage the model’s instruction following capabilities” (instruction-hierarchy-2404.13208.ro.txt, line 91). NIST generalises it: there is “an inherent trade-off between robustness and accuracy”, and “fully robust systems may be mathematically impossible without additional assumptions” (nist-ai-100-2e2025.txt, lines 936 and 2990). OWASP puts it practically: “it is unclear if there are fool-proof methods of prevention for prompt injection” (owasp-llm-top10-2025.txt, lines 249–250). A filter that lowers a hit rate is useful; calling it a boundary is the error.
Where the boundary actually is
Because no in-channel defence is complete, NIST advises designing “with the assumption that prompt injection attacks are possible if a model is exposed to untrusted input sources”, for example “by using multiple LLMs with different permissions” (nist-ai-100-2e2025.txt, lines 2944–2945). OWASP’s countermeasure is structural: run “adversarial testing and attack simulations”, “treating the model as an untrusted user to test the effectiveness of trust boundaries and access controls” (owasp-llm-top10-2025.txt, lines 281–283); “Require human approval for high-risk actions” (line 274). Google’s layered deployment — classifiers that “filter out harmful data containing malicious instructions” (google-indirect-injection.txt, line 73), a step that can “require user confirmation for certain actions” (line 82) — names its goal as “meaningfully elevating the difficulty, expense, and complexity faced by an attacker” (line 64): difficulty, not impossibility. The boundary is a decision the model cannot argue with, taken outside the text it reads — the enforcement sheet’s subject.
Level and prerequisites. L5 — the specialist level for attacking a running system and reading the result without over-claiming: design a falsifiable attack and tell a filter from a control. Prerequisites: the L4 sheets on the agent action loop, on direct and indirect prompt injection, on execution policy and on human approval; the L3 sheets on tool calling and argument validation.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Operations and risk — the node this sheet sits in.
- Companion sheets in this batch, not yet published: policy engines and enforcement outside the model; credential and permission lifecycle.
References
- Debenedetti et al. — AgentDojo — the tool-output hijack, the instructions-versus-data limit, and the ASR figures under different defences.
- Wallace et al. — The Instruction Hierarchy — the missing privilege boundary, the kernel-mode framing, and the over-refusal trade-off.
- Andriushchenko et al. — Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks — adaptive attacks, the measurement conditions behind an attack-success rate, and transfer.
- NIST — AI 100-2 E2025 — the data/instruction channel merge, the history of broken defences, the robustness trade-off, and the design assumption that injection is possible.
- OWASP — Top 10 for LLM Applications 2025, LLM01 Prompt Injection — the absence of fool-proof prevention and the adversarial-testing countermeasure.
- Google — Mitigating prompt injection attacks with a layered defense strategy — a deployed layered defence and its stated goal of raising attacker cost.