Jailbreak
Talking the model past its own restrictions.
A prompt crafted to get a model to produce output its safety training would normally prevent — role-play framing, obfuscation, incremental escalation.
Jailbreaks are an arms race. Build for the fact that some will succeed: assume you cannot prompt your way to safety and put the real controls elsewhere.
The direction of travel is discouraging in a specific way: as context windows grew, an attack that simply demonstrates the behaviour many times became feasible, which ties the attack surface to a capability trend rather than to a bug that can be patched.
The long context as an attack surface
A system accepts large amounts of user-supplied text.
Many-shot attacks scale with how much text you accept, so a generous input limit is a safety parameter, not just a cost one. Constraining input size and staying current with classifier defences are both parts of the response.
- blogMany-shot jailbreaking ↗
A long-context attack that uses many illustrative dialogues to steer a model past its guardrails — the link between context-window size and jailbreak surface.
- blogNext-generation Constitutional Classifiers ↗
Classifier-based jailbreak defence and its safety-usability trade-off: the earlier generation reduced jailbreak success but raised refusals on harmless queries.
- standardLLM01: Prompt injection ↗
Defines direct versus indirect injection and the downstream effects — data disclosure, unauthorised tool use, command execution — with least privilege and human approval as the recommended controls.
Prompt Injection
Malicious instructions hidden in content the model reads.
Refusal
The model declining a request.
Moderation
Filtering inputs and outputs against a policy.
Red Teaming
Adversarially trying to make the system do the wrong thing.