Articles

Trajectory replay and workflow regression

Replaying recorded trajectories to reproduce a workflow and catch behavioural change.

Reading: 4 minAutomation & AI

Article cover: Trajectory replay and workflow regression

A trajectory is what a workflow leaves behind: the ordered, correlated record of the operations — model calls, tool executions, retrieval, planning — that produced one outcome. Replaying it means re-feeding that record to re-run the workflow and asking whether it still behaves the same. This sheet owns that step alone: how signals correlate is the end-to-end observability sheet, how to compare results is the systematic-evaluation sheet, and where the check runs is the pipeline sheet.

Replay is not time travel. A workflow built on a language model is not deterministic, so “reproduce the workflow” cannot mean “re-obtain the same tokens”: it means reconstruct the same decision path and detect when a change to prompts, tools, models or orchestration makes it diverge.

A trajectory is a recorded trace

A distributed trace “records the path taken by a single request (made by an application or end user) as it propagates through multiple services in an architecture”, and “A trace is made of one or more spans.” OpenTelemetry frames a trace as “structured logs with context, correlation, hierarchy, and more baked in”. A trajectory is not a new artifact: it is a workflow’s spans.

Workflows get their own span: invoke_workflow “Represents an operation that executes a coordinated process composed of multiple agents or other operations involving generative AI”. Inside it, each tool execution is a span whose “gen_ai.operation.name SHOULD be execute_tool”. That parent-child hierarchy orders the record for replay.

What a trajectory must contain to be replayable

To replay a workflow you must reconstruct, for each step, what went in and what came out. The conventions mark most of that payload opt-in: gen_ai.tool.call.arguments (“Parameters passed to the tool call”) and gen_ai.tool.call.result (“The result returned by the tool call”) are Opt-In, as are gen_ai.input.messages and gen_ai.tool.definitions (“The list of tool definitions available to the GenAI agent or model”). The default is plain: “OpenTelemetry instrumentations SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in.”

That default is the central decision. A trajectory recording only structure — span names, hierarchy, model and workflow name — gives the shape of what happened; one that also records content can be replayed step by step. The production pattern is to “Store content externally and record references on the spans”. Capturing everything is costly and sensitive; capturing nothing leaves nothing to re-feed. A span is also a logical operation: it covers a retried request “with all retries”, so the trajectory stores the call, not the attempts.

Determinism, and its absence

Ordinary code assumes one input, one output; a model call does not. The seed attribute is hedged exactly as reality is — requests with the same seed are “more likely to return same result”, not guaranteed — and model revisions and sampling move the output anyway.

Two modes of replay: re-execute or substitute

Replay therefore splits into two modes, and a serious design picks one. Re-executing the model runs the real calls and lets them vary: it tests the system end to end, but confounds your change with sampling noise. Substituting the recorded outputs pins the model’s answers and re-runs only the orchestration: it isolates the decision path, but stops testing the model.

Orchestration state is where replay comes closest to deterministic — and even there it is not free. A checkpointing framework persists “a thread’s graph state as checkpoints”, and “the checkpointer writes the exact graph state so you can resume later”. Resuming is not silent: “The node restarts from the beginning of the node where the interrupt was called when resumed, so any code before the interrupt runs again”. Re-running a node re-runs its side effects; a replay that re-issues a payment is an incident.

What a workflow regression test can and cannot prove

What it can prove is that one observed failure becomes permanent. “Add real traces to a dataset so a failure you saw once becomes a test you run every time”, and evaluation “scores your agent’s outputs against datasets and criteria so you can benchmark versions, catch regressions, and track quality over time”. On a fixed set of recorded inputs you can assert the path did not change: same tools, same order, same termination — even when the words differ.

What it cannot prove is anything about trajectories you did not record. A dataset of pinned runs is a sample, not a specification: it cannot show the workflow is correct on inputs it has never seen. And a divergence may not be a regression at all — a tool returned a different document on the day of replay, the model behind the same name was updated, or a seed that only made repetition likely did not hold. Attributing a divergence to your change needs everything else held fixed, which for a hosted model you cannot fully do: read it as a question — “what changed?” — not a verdict.

Level and prerequisites. L5 — advanced systems engineering: the reader must design a replay harness, decide what to capture, and choose between re-executing and substituting the model. Prerequisites: the L1–L2 foundations of this area, the L3 evaluation sheets, and the L4 sheets on agent architecture, checkpoints and long-running workflows.

Where to go next

References

  • OpenTelemetry — Traces — the framing of a trace as structured logs with context, correlation and hierarchy, and the span fields.
  • OpenTelemetry — Observability primer — the distributed-trace definition and the parent-child waterfall.
  • OpenTelemetry — Semantic conventions for generative client AI spans — the execute_tool span, the opt-in capture of content, and the retry-span semantics.
  • OpenTelemetry — Semantic conventions for GenAI agent and framework spans — the invoke_workflow span, the seed attribute, and tool definitions.
  • LangGraph — Persistence — checkpointers, time travel, fault tolerance, and the in-memory limit.
  • LangGraph — Interrupts — the exact-state resume and the node that restarts when resumed.
  • LangSmith — Evaluation — traces promoted to datasets, and regression testing over versions.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.