Agent Evaluations in Real Environments with Tracevell

Agent evaluations test your agent end to end, the way it runs in production. Every eval case gets its own isolated container: your real application with its real services (database, queue, cache) and the data that reproduces one specific scenario. A driver calls the app through an entry point it already has, and a verifier scores the artefacts of the run. You run the evals with lmnr-cli eval run, locally on Docker or in Tracevell’s cloud sandboxes, and every trial lands in Tracevell with its score next to the full trace of the agent run.

​

Agent evaluations and LLM evaluations

Tracevell has two ways to evaluate, based on what the evaluations look at:

• Agent evaluations (this section) set up a working environment for the agent, let it run, and score the whole outcome, including the state changes, and the agent trajectory.
• LLM evaluations call a function in your process with evaluate(), then compare its output to target data with evaluator functions.

LLM evaluations work for agents too, and for a prompt, a classifier, or a short pipeline they are often the fastest option. Agent evaluations fit once the agent uses tools and acts on real systems. The two share the same UI views, run comparison, and SQL access. Both can also coexist in one project.

​

Why evaluate the agent in its environment

Unit tests are a poor fit for agents that act on the world:

• The outcome is not in the final message. Agents may write database rows and change the state of the world around them. Scoring just the final message misses whether the right thing actually happened.
• The bugs are in the integration. Downstream services may experience transient issues; agents may or may not self-heal. Fixtures hide these kinds of issues.
• Production failures need their exact context. Reproducing a failure means the same conversation, records, and account state.

Agent evaluations answer all three by giving every case a fresh, complete copy of the stack: your real code and services, seeded with the case’s data.

• Isolation. Each case boots its own stack with its own data, so cases don’t interfere with each other and can run in parallel, in any order.
• Local and cloud from one folder. lmnr-cli eval run runs trials on your Docker. Add --remote to run the same eval in Tracevell sandboxes, one per trial.

​

How a run works

You keep an evals/ folder next to your code. Each case in it is a Harbor task: image definition, the data for this scenario, and a verifier. When you run lmnr-cli eval run, Tracevell:

1. Builds your image from the live source tree.
2. Starts one container per trial, uploads the case’s data, and runs the driver against your app.
3. Runs the verifier over what the app produced and records its reward as the score.
4. Attaches the app’s LLM and tool calls to the trial’s trace, so you can investigate failing scores.

Every run of a suite is charted in one evaluation group. Concepts shows what goes into a trial’s container and what comes out, and maps each piece to what you see in Tracevell.

​

Next steps