Articles

Managing context in long-running workflows

A long workflow outgrows its window: what stays, what is compacted, what must survive.

Reading: 4 minAutomation & AI

Article cover: Managing context in long-running workflows

An LLM is handed a set of tokens and returns a response; it keeps nothing between calls. A long-running workflow is therefore a sequence of turns that must each re-supply the state the model needs, and that set of tokens — the context — does not grow. It either fits, or it is managed. Context management is the engineering decision of what stays in the window as a workflow runs on: which text is summarised away, which is evicted, which is pushed outside and reloaded, and which must survive every pass untouched.

Anthropic states the scope: long-horizon tasks are those “where the token count exceeds the LLM’s context window”. This sheet is about the window — the managed resource — not the model that reads it.

Why a long workflow exhausts its context

The window is a budget, not a size. Anthropic reports that models “lose focus or experience confusion at a certain point”, and that “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases”; every new token, it adds, “depletes this budget by some amount”. Adding history is not neutral: more text can make each existing item harder to use. The long-context study measured the same effect in position rather than length: performance is highest when the relevant passage sits “at the beginning or end of the input context” and degrades “in the middle of long contexts”, “even for explicitly long-context models”. A workflow that appends every tool result pushes real evidence toward the region where it is used least.

The levers that change the window

Anthropic names four ways a long task works around the limit — compaction, structured note-taking, sub-agent architectures, and just-in-time retrieval — while the memory documentation lists the same family for a single thread: trim the oldest messages, delete them, or summarize them. Compaction replaces the history with a model-written summary and continues. Tool-result clearing is the lightest form: “One of the safest lightest touch forms of compaction is tool result clearing”. Structured note-taking keeps the summary outside the window and pulls it back later. Just-in-time retrieval keeps only “lightweight identifiers” and loads data at runtime; Anthropic notes the cost: “runtime exploration is slower than retrieving pre-computed data”. A sub-agent runs its own long search in a clean window and returns “only a condensed, distilled summary”.

Compaction is a lossy decision

Compaction is the primary lever and also the riskiest, because a summary is a bet about the future. Anthropic is explicit that “overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later”. Its tuning advice follows from that: build the compaction prompt to maximize recall first, then improve precision by cutting what is superfluous. The production implementation preserves “architectural decisions, unresolved bugs, and implementation details” while discarding redundant tool output. The question is therefore not how to shorten, but which facts later steps cannot re-derive.

What must never be dropped

Some content is not history and must survive every pass. The request has a fixed frame: the provider orders the prompt as “tools, system, and messages (in that order)”, and the system instructions and tool definitions against which every step is read belong to that frame, not to the compactable conversation. There are also structural invariants of the message list itself. Vendor documentation warns that after any deletion the history must stay valid — “Some providers expect message history to start with a user message”, and “Most providers require assistant messages with tool calls to be followed by corresponding tool result messages”. Summarizing away a tool result, or dropping the leading user turn, leaves a history the API may reject outright. The safe rule is to compact the conversation and never the skeleton around it.

Where the trade-off is decided

No single lever wins. A summary buys room at the cost of fidelity; note-taking buys durability at the cost of a reload step; a sub-agent buys a clean window at the cost of coordination and a lossy hand-back. Which to use depends on the shape of the task — Anthropic ties compaction to back-and-forth work, note-taking to iterative work with clear milestones, and sub-agents to parallel exploration. The mechanics of each live in neighbouring sheets: the memory store is the short- and long-term memory sheet’s, run-state durability is the checkpointing sheet’s, and prefix reuse is the caching sheet’s. This sheet owns only the question it opened with — what remains in the window.

Level and prerequisites. L5 — the reader must choose an eviction-and-compaction policy for a system that keeps running past its context limit, and must know which content is invariant under that policy. It is systems-level engineering, not the API of one component. Prerequisites: the L1 sheet on the context window and its practical limits, the L2 sheets on context, throughput and serving, the L3 sheet on context assembly, and the L4 sheets on short- and long-term memory and on state, checkpoints and resuming.

Where to go next

References

  • Anthropic — Effective context engineering for AI agents — context rot, the attention budget, the long-horizon techniques, the compaction definition and its recall-versus-precision tuning, and the fixed order of the prompt.
  • Anthropic — Prompt caching — that the prompt is ordered tools, system, messages.
  • Liu et al. — Lost in the Middle — where in a long input relevant information is actually used.
  • LangChain — Memory (LangGraph documentation) — the trim/delete/summarize levers and the message-history validity constraints.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.