Vol. XXV · Tools and Agents Plate 8 of 9agent safety, permissions, sandboxing, human confirmation, prompt injection defences, least privilege, untrusted dataCE

Agents and Risk.

Give it only the access the task needs, run it where mistakes are cheap, and ask before anything irreversible.

Answers can be wrong; actions can be wrong and irreversible. An agent with a shell can delete a directory, an agent with email can send a message, an agent with a payment tool can spend. Harm comes from two sources: the agent’s own mistakes, and injected instructions — text in a web page, file or tool result, written by someone other than the user, that the model reads and mistakes for instructions, because to a language model everything in the context is text (plate XXV.2). This plate is about defences; it shows no attacks.

The defences come from computer security. Least privilege: give the agent only the tools and permissions the task needs. Sandboxing: run it in an isolated environment where damage is contained and reversible. Confirmation: require a human to approve consequential actions. Separating data from instructions: designs in which content the agent reads cannot change what it is allowed to do — for example, a planner that sees only the user’s request decides the allowed actions before any untrusted content is read (CaMeL, Debenedetti et al., 2025). The workshop models 1 000 tasks with the chosen exposure to untrusted content and the chosen defences, and counts harmful actions that get through.

Below: expected harmful actions per 1 000 tasks with each defence added in turn; readouts give the total with the chosen defences and the confirmations asked of a person.

What you will learn
Workshop
A model of layered defences, not a measurement. 1 000 tasks; per task, an own-mistake risk m = 0.5% of a harmful action, plus a chance q of meeting untrusted content times a chance f = 20% that the model is misled by it. Least privilege leaves fraction a_priv = 0.2 of harmful actions possible. Confirmation catches fraction c = 0.9 of what reaches it and asks k = 2 times per task. Data–instruction separation blocks fraction s = 0.95 of misled actions (not own mistakes). Sandboxing makes fraction 0.8 of remaining harms reversible.

Layers multiply

With half of all tasks reading untrusted content and no defences, the model predicts about 105 harmful actions per 1 000 tasks — most from being misled, a few from the agent’s own mistakes. Least privilege, confirmation, separation and a sandbox each remove a share of what is left, and because they act on different things their effects multiply: with all four the expected count falls to 0.040. The price of confirmation is interruptions — 400 approvals per 1 000 tasks with least privilege narrowing what needs asking, against 2,000 without.

No single layer suffices. Training models to resist misleading content lowers f but does not make it zero; confirmation fails if people approve without reading, which they do when asked too often; sandboxes do not protect anything the sandbox can reach, such as credentials placed inside it. The durable designs make the safe path structural — permissions decided before untrusted content is read, consequential actions gated outside the model — rather than relying on the model to always notice. Vol. XXX returns to how agent risks are assessed at the frontier.

Treat everything an agent reads — web pages, documents, emails, tool results — as untrusted data from a stranger. The user’s request is the only instruction; everything else is information.

Terms

Injected instruction
Text in content the agent reads that tries to direct it.
Least privilege
Granting only the permissions a task needs.
Sandbox
An isolated environment where actions are contained and reversible.
Confirmation
Human approval before a consequential action.
Agent risk comes from mistakes and misleading content; layered, structural defences multiply, and no single one is enough.

Sources

  1. Greshake et al., “Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection”, AISec 2023.
  2. Debenedetti et al., “Defeating Prompt Injections by Design” (CaMeL), arXiv:2503.18813, 2025.
  3. Saltzer and Schroeder, “The Protection of Information in Computer Systems”, Proceedings of the IEEE, 1975 (least privilege).