Red Teaming
Adversarially trying to make the system do the wrong thing.
Deliberately probing for failures: prompt injection, jailbreaks, harmful outputs, unsafe tool use, data exposure. Done before launch and repeated as the system changes.
For agents, red teaming extends beyond text to actions: can it be convinced to delete, exfiltrate or spend? The blast radius of a successful attack is the environment you gave it.
Automate what can be automated and keep people for the rest. Generative eval frameworks can produce large volumes of adversarial scenarios, but a person who wants to break your specific system will find things a generator does not think of. Treat the result as a starting inventory of weaknesses, and rerun it whenever the system changes.
Attacking the agent, not just the model
Preparing a tool-using agent for launch.
Include indirect injection through retrieved documents, an attempt to exfiltrate data via a tool call, and a check that a destructive action still requires approval. Model-level jailbreak testing alone will not have covered any of those.
- blogIntroducing Bloom: an open source tool for automated behavioural evaluations ↗
An agentic framework that generates behavioural eval scenarios and scores them with model judges, with a human-calibration check and a warning that eval designs go stale.
- standardLLM01: Prompt injection ↗
Defines direct versus indirect injection and the downstream effects — data disclosure, unauthorised tool use, command execution — with least privilege and human approval as the recommended controls.
- docsSafety best practices ↗
Application-level controls: adversarial testing, human review for high-stakes and code output, moderation, bounded inputs, and telling users what the model cannot do.
Prompt Injection
Malicious instructions hidden in content the model reads.
Jailbreak
Talking the model past its own restrictions.
Guardrails
Constraints on what the system will do, enforced in code.
Evaluation · Evals
How you know the system is actually working.
Sandbox
An isolated environment where an agent can act without risk to real systems.