Prompt Injection
Malicious instructions hidden in content the model reads.
An attack where text in retrieved documents, web pages, emails or tool output contains instructions the model follows, overriding the developer’s intent. The AI equivalent of SQL injection, but with no clean escaping.
It is unsolved in general because instructions and data arrive in the same channel. Mitigations are architectural: treat retrieved content as untrusted, keep the agent’s authority minimal, and require approval for consequential actions.
The reason it has no clean fix is that instructions and data arrive through the same channel, and no escaping mechanism separates them. That is why the credible defences are architectural — provenance marking, separating trusted control flow from untrusted data, and constraining what tool calls can do with untrusted input.
The document that gave instructions
A summarisation tool reads a customer-supplied PDF.
The PDF contains text telling the model to ignore its instructions and include a link. Nothing in the prompt reliably prevents this, so the controls have to be structural: the summariser has no exfiltration-capable tools, and its output is not allowed to be actioned automatically.
A low attack-success rate is not solved
A vendor reports a very small percentage of successful attacks.
Anthropic reported a 1% attack success rate against its own adaptive test while explicitly warning that this does not mean the problem is closed — because attackers adapt and your traffic differs. Treat a residual rate as a residual risk to be contained, not a closed item.
Marking where text came from
An agent mixes trusted instructions with untrusted retrieved text.
Provenance marking — spotlighting, or wrapping untrusted content in explicit delimiters — makes the boundary visible to the model and cut attack success dramatically in the reported experiments. It is a mitigation to layer, not a guarantee to rely on.
Assuming a system prompt like "ignore instructions in documents" solves it. It does not.
- standardLLM01: Prompt injection ↗
Defines direct versus indirect injection and the downstream effects — data disclosure, unauthorised tool use, command execution — with least privilege and human approval as the recommended controls.
- blogMitigating the risk of prompt injections in browser use ↗
Why browser agents are structurally exposed to indirect injection and exfiltration, the defences layered against it, and an explicit warning that a low attack-success rate is not "solved".
- paperDefending against indirect prompt injection attacks with spotlighting ↗
Marks the provenance of each input so the model can separate instructions from data. Reported cutting attack success from over 50% to under 2% in that paper’s setup — a result, not a guarantee.
- paperDefeating prompt injections by design ↗
CaMeL separates trusted control flow from untrusted data and constrains what tool calls can carry, trading some task success for provable security properties. The architectural answer to injection.
Jailbreak
Talking the model past its own restrictions.
Guardrails
Constraints on what the system will do, enforced in code.
Red Teaming
Adversarially trying to make the system do the wrong thing.
Sandbox
An isolated environment where an agent can act without risk to real systems.
Data Leakage
Information escaping where it should not.