Vol. XXV · Tools and Agents Plate 4 of 9context engineering, context window, summarisation, compaction, truncation, memory, long tasksCE

Managing the Context.

The window is finite and every token in it is paid for at every step. Choose what stays.

An agent’s context grows with every turn: the instructions, the tool descriptions, and then each action and observation in turn. A long coding or research session produces far more text than any window holds, and even within the window, every decode step reads every cached token (plate XXIII.5), and the model’s use of information in the middle of long contexts is imperfect (plate VI.8). Context engineering is deciding what the window contains at each step.

Three policies bracket the choices. Keep everything until the window is full, then fail. Truncate the oldest turns, keeping a fixed recent window — cheap, but whatever the task needed from early turns is gone. Summarise: when the context reaches a threshold, replace older turns with a short summary written by the model, at a small risk per summary of dropping something that mattered. The workshop simulates an agent of the chosen number of turns, each adding the chosen number of tokens, against a window of the chosen size, where the task needs one fact from an early turn.

Below: context length at each turn under the three policies, against the window; readouts give tokens read over the whole task and the chance the needed early fact is still available at the end.

What you will learn
Workshop
Simulated: a system prompt of 2 000 tokens, then n turns each adding the chosen number of tokens; the needed fact arrives in turn 2. Truncation keeps the system prompt and the most recent turns that fit in half the window. Summarisation replaces all but the last 4 turns with a 1 000-token summary whenever the context passes 80% of the window; each summary drops the fact with probability ε = 5%. Tokens read: the context length summed over turns (each step re-reads its context).

Pay for every token at every step

With 100 turns of 1,500 tokens and a 128K window, keeping everything would need 152,000 tokens of context by the end — more than the window holds — and would re-read 7.78 million tokens over the task, because every step reads the whole context so far. Truncating to half the window keeps the cost down (5.12 million tokens read) but drops turn 2 long before the end. Summarising whenever the context passes 80% keeps the window bounded, reads 4.65 million tokens, and keeps the early fact with probability 95% after its summaries (1 here) — the risk compounding with each one.

Practical agents combine these with two more tools. Prefix caching (plate XXIII.6) makes re-reading the stable front of the context cheap, which favours keeping instructions fixed and appending. External memory — files, notes, a scratchpad the agent writes and later reads back — moves facts out of the window entirely and retrieves them when needed (plate XXV.3). Choosing what to keep is now treated as an engineering discipline of its own, because for long tasks it matters as much as the model.

Tool outputs are the usual culprit: a single command can print thousands of lines. Truncating or summarising observations at the harness, before they enter the context, is often the cheapest fix.

Terms

Context window
The maximum tokens the model can attend to at once.
Truncation
Dropping the oldest turns to fit.
Compaction
Replacing older turns with a model-written summary.
External memory
Notes or files the agent writes and reads back instead of keeping everything in context.
Agent contexts grow every turn and are re-read every step; truncation forgets, summaries risk detail, and external memory moves facts out of the window.

Sources

  1. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts”, TACL 2024.
  2. Packer et al., “MemGPT: Towards LLMs as Operating Systems”, arXiv:2310.08560, 2023.
  3. Anthropic Engineering, “Effective context engineering for AI agents”, 2025.