Skip to main content
AI agents often fail without throwing an error. They cite the wrong document, click the wrong button, or present an unsupported claim as fact. These mistakes are an inherent risk of relying on probabilistic models, yet a run can look healthy to traditional observability systems: every tool call succeeds, latency stays normal, and no errors are logged. An alert based on those metrics cannot tell you that a legal agent cited the wrong document. Detecting these failures requires understanding what the agent was asked to do and how it went about doing it. That evidence lives in its trace: the inputs, recorded reasoning, tool calls, and outputs that show how the run unfolded. Tracevell’s Signals analyze agent traces to identify failures and turn them into structured events you can monitor and alert on.

​
From traces to structured events

Signals are powered by our trace analysis agent. The Signals agent takes three inputs:
  • Trace: the agent run to analyze.
  • Instruction: what to look for, in plain language.
  • JSON schema: what to record when it finds a match.
A trace and a Signal's instruction and schema go into the Signals agent, which outputs a signal event. In the trace, a coding agent runs tests: one test fails on a missing function, the end-to-end test fails on a different error, and the unit tests pass. Its final report blames both failures on the same missing function. The instruction asks whether the agent's final report states a conclusion that its own tool outputs contradict, with a schema of claim, evidence and claim_type. The Signals agent outputs an event: claim_type root_cause, evidence 'e2e test: different error', severity critical.

A coding agent's report contradicts its own test output, and the Signals agent records it as a structured event.

An instruction and a schema together define a Signal. The Signals agent reads the relevant steps in each trace and returns a signal event with your schema fields and links to its evidence. If it finds no match, it returns no event.
The lifecycle of a Signal run. A trace finishes and is checked against the Signal's trigger, sample rate and filters; a trace that fails them is skipped and no run is recorded. A trace that passes goes to the Signals agent; if it finds no match, the run completes with no event. If it finds a match, it returns a signal event with its trace_id, claim_type root_cause and severity critical. The event joins a cluster, is stored in the signal_events table for SQL, and is checked against alert rules, which notify Slack or email on live runs.

Each finished trace is checked against the Signal's trigger and filters, then read by the Signals agent. A matching trace produces an event that is grouped into a cluster, stored for SQL, and checked against your alerts.

To set when Signals agent starts processing your trace you can configure triggers and trace filters, or use a backfill to analyze historical traces. Once a signal event is created, its structured fields are available to query in SQL. You can filter events, count how often a behavior occurs, and compare outcomes across traces. Tracevell automatically groups similar events into clusters based on the meaning of their findings. Clusters help you spot patterns across thousands of events, see which issues recur, and understand how common they are. On top of that, you can configure alerts to notify you in Slack or by email when a new signal event matches your conditions or a new cluster forms.

​
Example: catch a report that contradicts its own evidence

In the trace above, a coding agent reports its work as done despite two failing tests. It blames both failures on the same pre-existing problem, but one test log shows a different error. Every tool call succeeded. The mistake is in the agent’s reasoning, and if you trust its report, you’ll look for the wrong bug. To find reports that contradict the agent’s own evidence, we created a Signal with the following instruction:
Does the agent’s final report state a conclusion that its own tool outputs contradict? Record the claim and the evidence against it.
The schema defines three fields:
  • claim: the statement in the final report.
  • evidence: the tool output that contradicts it.
  • claim_type: one of root_cause, completion, verification, or other.
For this trace, the event looks like this:
Now, we can use claim_type across traces to see which kinds of claims our agent gets wrong most often.

​
Cost efficient and reliable analysis on every trace

To monitor an agent in production, trace analysis needs to be accurate enough to trust and cheap enough to run on every trace. Sampling leaves failures unseen, while noisy results make alerts hard to act on. Off-the-shelf models often force a tradeoff: cheaper models miss failures or flag problems the evidence doesn’t support, while more capable models make analyzing every trace too expensive at production scale. We trained flow-1 with reinforcement learning specifically for agent trace analysis to meet both requirements:
  • Quality: it matches GPT-6-sol in failure detection quality on our benchmark, while being 23x cheaper.
  • Cost: It would cost just $3.5 on average to process 1k traces of 100k tokens each. Learn more about Signals pricing here, and cap your spend with usage warnings and hard limits.
flow-1 gives GPT-6-sol level trace analysis with 23 times more traces processed per $1. One dot per trace analyzed for $1: GPT-6-sol analyzes 38 traces, flow-1 analyzes 888.

One dot per trace analyzed for $1.

That makes it practical to analyze every production trace with the quality needed to catch subtle failures.

​
Define your own trace analysis

Signals are a general way to analyze traces. You decide what to look for and what to record. An instruction can ask a broad question, combine several conditions, or compare evidence across different steps. The output schema gives you control over the fields and categories in the result. For example, you could ask a Signal to:
  • Detect a specific behavior: Did the agent promise a refund without calling the refund tool? Record the promise and the missing action.
  • Classify a conversation: What was the user trying to accomplish, and was it resolved? Return an intent and an outcome.
  • Extract information: Which integrations did the user ask for? Collect the integration names and the requests they made.
  • Compare evidence: Do the amounts in the contract agree with the accompanying memo? Record any mismatches and where they appear.

​
Write the instruction

The instruction is not a full LLM prompt. The Signals agent already knows that it is analyzing a trace, how the trace is laid out, how to cite the spans behind a finding, and how to return an event in your schema, or no event when nothing matches. Your instruction only says what to look for. A general description is usually enough. These are complete instructions:
Find runs where the agent gave up before completing the user’s request.
Did the agent promise a refund without calling the refund tool?
What was the user trying to accomplish? Classify the primary intent.
Leave out what the Signals agent already handles:
  • A role or persona. Don’t open with You are an expert judge evaluating AI agents. The agent’s role is fixed.
  • How to read the trace. The trace format, span types, and long inputs and outputs are handled for you.
  • The output format. Don’t ask for JSON or describe the response shape. Put the fields you want in the output schema, and use each field’s description to say what goes in it.
  • When to stay quiet. A trace with no match already returns no event.
Add detail only when the results need it. If events include cases you don’t consider matches, say what does and doesn’t count, as in Report tool calls with invalid arguments. Do not report tool calls that failed because of a network error. The Failure Detector template in the Create Signal drawer is a longer example: it spells out what is reportable and what is not, and still leaves the role and output format to the Signals agent. Clusters show you where an instruction is too broad.

​
How Signals are billed

On Tracevell Cloud, Signals run on Tracevell’s own model, so you don’t bring a model API key. Every Signal run is billed by the tokens the Signals agent uses to read the trace and write its result (input, cached input, and output tokens).
  • What counts: every run, from live triggers and from backfills. A trace that doesn’t pass the Signal’s trigger, filters, or sample rate is skipped and costs nothing.
  • Credit: every new workspace gets a one-time $5 Signals credit. On the Free plan, Signals stop when the credit is used up. Paid plans bill usage beyond the credit per token.
  • Rates: per-token prices for each plan are on the pricing page.
  • Control spend: filters and the sample rate decide how many traces each Signal analyzes. Track spend in Settings > Usage and cap it with usage warnings and hard limits.

​
Get started

If you’re new to Tracevell, start by sending your agent’s traces. Every new project includes a Failure Detector Signal. Open Signals in your project to inspect its findings. When you create your own Signal, review its findings on representative traces before relying on its alerts. The results are model judgments, and the linked evidence lets you check them.

Quickstart

Create your first Signal and inspect its events.

CLI

Define and manage Signals from your terminal.

Clusters

Explore recurring patterns across traces.

Alerts

Get notified when a Signal finds an issue or a new event cluster forms.