Trajectory replay and workflow regression
Replaying recorded trajectories to reproduce a workflow and catch behavioural change.

A trajectory is what a workflow leaves behind: the ordered, correlated record of the operations — model calls, tool executions, retrieval, planning — that produced one outcome. Replaying it means re-feeding that record to re-run the workflow and asking whether it still behaves the same. This sheet owns that step alone: how signals correlate is the end-to-end observability sheet, how to compare results is the systematic-evaluation sheet, and where the check runs is the pipeline sheet.
Replay is not time travel. A workflow built on a language model is not deterministic, so “reproduce the workflow” cannot mean “re-obtain the same tokens”: it means reconstruct the same decision path and detect when a change to prompts, tools, models or orchestration makes it diverge.
A trajectory is a recorded trace
A distributed trace “records the path taken by a single request (made by an application or end user) as it propagates through multiple services in an architecture”, and “A trace is made of one or more spans.” OpenTelemetry frames a trace as “structured logs with context, correlation, hierarchy, and more baked in”. A trajectory is not a new artifact: it is a workflow’s spans.
Workflows get their own span: invoke_workflow “Represents an operation that executes a coordinated process composed of multiple agents or other operations involving generative AI”. Inside it, each tool execution is a span whose “gen_ai.operation.name SHOULD be execute_tool”. That parent-child hierarchy orders the record for replay.
What a trajectory must contain to be replayable
To replay a workflow you must reconstruct, for each step, what went in and what came out. The conventions mark most of that payload opt-in: gen_ai.tool.call.arguments (“Parameters passed to the tool call”) and gen_ai.tool.call.result (“The result returned by the tool call”) are Opt-In, as are gen_ai.input.messages and gen_ai.tool.definitions (“The list of tool definitions available to the GenAI agent or model”). The default is plain: “OpenTelemetry instrumentations SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in.”
That default is the central decision. A trajectory recording only structure — span names, hierarchy, model and workflow name — gives the shape of what happened; one that also records content can be replayed step by step. The production pattern is to “Store content externally and record references on the spans”. Capturing everything is costly and sensitive; capturing nothing leaves nothing to re-feed. A span is also a logical operation: it covers a retried request “with all retries”, so the trajectory stores the call, not the attempts.
Determinism, and its absence
Ordinary code assumes one input, one output; a model call does not. The seed attribute is hedged exactly as reality is — requests with the same seed are “more likely to return same result”, not guaranteed — and model revisions and sampling move the output anyway.
Two modes of replay: re-execute or substitute
Replay therefore splits into two modes, and a serious design picks one. Re-executing the model runs the real calls and lets them vary: it tests the system end to end, but confounds your change with sampling noise. Substituting the recorded outputs pins the model’s answers and re-runs only the orchestration: it isolates the decision path, but stops testing the model.
Orchestration state is where replay comes closest to deterministic — and even there it is not free. A checkpointing framework persists “a thread’s graph state as checkpoints”, and “the checkpointer writes the exact graph state so you can resume later”. Resuming is not silent: “The node restarts from the beginning of the node where the interrupt was called when resumed, so any code before the interrupt runs again”. Re-running a node re-runs its side effects; a replay that re-issues a payment is an incident.
What a workflow regression test can and cannot prove
What it can prove is that one observed failure becomes permanent. “Add real traces to a dataset so a failure you saw once becomes a test you run every time”, and evaluation “scores your agent’s outputs against datasets and criteria so you can benchmark versions, catch regressions, and track quality over time”. On a fixed set of recorded inputs you can assert the path did not change: same tools, same order, same termination — even when the words differ.
What it cannot prove is anything about trajectories you did not record. A dataset of pinned runs is a sample, not a specification: it cannot show the workflow is correct on inputs it has never seen. And a divergence may not be a regression at all — a tool returned a different document on the day of replay, the model behind the same name was updated, or a seed that only made repetition likely did not hold. Attributing a divergence to your change needs everything else held fixed, which for a hosted model you cannot fully do: read it as a question — “what changed?” — not a verdict.
Level and prerequisites. L5 — advanced systems engineering: the reader must design a replay harness, decide what to capture, and choose between re-executing and substituting the model. Prerequisites: the L1–L2 foundations of this area, the L3 evaluation sheets, and the L4 sheets on agent architecture, checkpoints and long-running workflows.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Operations & risk — the node this sheet sits in.
References
- OpenTelemetry — Traces — the framing of a trace as structured logs with context, correlation and hierarchy, and the span fields.
- OpenTelemetry — Observability primer — the distributed-trace definition and the parent-child waterfall.
- OpenTelemetry — Semantic conventions for generative client AI spans — the execute_tool span, the opt-in capture of content, and the retry-span semantics.
- OpenTelemetry — Semantic conventions for GenAI agent and framework spans — the invoke_workflow span, the seed attribute, and tool definitions.
- LangGraph — Persistence — checkpointers, time travel, fault tolerance, and the in-memory limit.
- LangGraph — Interrupts — the exact-state resume and the node that restarts when resumed.
- LangSmith — Evaluation — traces promoted to datasets, and regression testing over versions.