Agent Evaluations Quickstart with lmnr-cli

This quickstart gets a first agent evaluation suite running with lmnr-cli eval run. The fastest path is to let your coding agent build it with the Tracevell skill. If you would rather see every file, follow Get started manually below.

Set up with your coding agent

You need Docker, uv, and Node.js. From the repo root, link the directory to a Tracevell project:

npx lmnr-cli@latest setup

setup signs you in, writes LMNR_PROJECT_API_KEY to .env, and installs the Tracevell agent skill, which knows the evals folder layout. Then prompt your coding agent. For example:

Set up agent evaluations for the refund flow with lmnr-cli eval run. Use our real database and seed each case from a production trace.

If you don’t have production traces yet, describe in plain words what you want to evaluate. The skill follows the same guidance as these docs: it finds the entry point, builds one image, seeds per-case data, writes the verifier, and runs the suite locally. Review what it wrote against Build a production harness, then open the run in Tracevell.

Get started manually

This walkthrough evaluates a support agent that handles billing tickets. It looks up a customer’s charges in a SQLite database and can refund one. Each case starts with its own copy of the database, and the verifier checks two things: whether the right refund is in the database, and whether the reply to the customer is any good. An LLM judge grades the reply. The complete project is examples/agent-evals in the lmnr-python repo. Clone it to follow along, or copy the files into your own repo as you go. The agent is a single module, app/support_agent.py. It runs an Anthropic tool loop over two tools, list_charges and refund_charge. It calls Laminar.initialize() and wraps the tools in observe, so each trial’s trace shows the model calls and the tool calls. Trimmed to the parts the harness relies on:

app/support_agent.py

import os

from lmnr import Laminar, observe

Laminar.initialize()

DB_PATH = os.environ.get("APP_DB", "/srv/app/app.db")
MODEL = os.environ.get("AGENT_MODEL", "claude-sonnet-5")

@observe(span_type="TOOL")
def list_charges(email: str) -> list[dict]: ...

@observe(span_type="TOOL")
def refund_charge(charge_id: str) -> dict: ...

@observe()
def handle_ticket(ticket: str) -> str:
    ...  # call the model, run the tools it asks for, return its reply

The driver imports DB_PATH and handle_ticket, and the Harbor agent sets AGENT_MODEL. The app’s dependencies, anthropic and lmnr, are in the project’s pyproject.toml and uv.lock, which the image installs.

Install the prerequisites and link the project

You need Docker, uv, and Node.js for lmnr-cli. lmnr-cli eval run runs a pinned version of Harbor with the Tracevell plugin through uvx, so you don’t install Harbor yourself.From the repo root, link the directory to a Tracevell project. Skip this if you already ran it above:

npx lmnr-cli@latest setup

setup signs you in and writes LMNR_PROJECT_API_KEY to .env. See Tracevell CLI for details.

Lay out the evals folder

Create this structure next to your code:

evals/
  lmnr.yaml                     # the image tasks use, and how to build it
  image/
    Dockerfile
    shared/
      run_case.py               # seeds the database, calls the agent, writes result.json
  support/                      # a suite
    harbor.yaml
    agent/
      __init__.py               # empty
      agent.py
    tasks/
      duplicate-charge/         # a case
        task.toml
        instruction.md
        environment/
          case.json             # the ticket
          seed.json             # the database rows for this case
        tests/
          test.sh               # the verifier
          verify.py
          expected.json         # the reference answer

Describe the image

lmnr.yaml names the image and says how to build it. Paths are relative to evals/, and context: .. makes the repo root the build context, so the image is built from your live source:

evals/lmnr.yaml

images:
  refund-agent-evals:
    dockerfile: image/Dockerfile
    context: ..

The Dockerfile starts from uv’s official image, which already has uv and Python 3.14 on Debian trixie. It installs the locked dependencies first, so they stay cached when only the app code changes:

evals/image/Dockerfile

FROM ghcr.io/astral-sh/uv:python3.14-trixie-slim

ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
WORKDIR /srv/app

# Dependencies first, so they stay cached when only the app code changes.
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-install-project
ENV PATH="/srv/app/.venv/bin:$PATH"

COPY evals/image/shared/ /opt/harness/
COPY app/ ./app/

WORKDIR /case

The driver creates this case’s database from seed.json, imports the production function unchanged, calls it with the ticket, and writes the outcome to /logs/agent/result.json:

evals/image/shared/run_case.py

import json
import sqlite3
import sys
import time

sys.path.insert(0, "/srv/app")
from app.support_agent import DB_PATH, handle_ticket  # the production code path

case = json.load(open("/case/case.json"))

# Create this case's database from its seed data.
conn = sqlite3.connect(DB_PATH)
conn.executescript(
    """
    CREATE TABLE charges (
        id TEXT PRIMARY KEY, email TEXT, description TEXT, amount REAL, created_at TEXT
    );
    CREATE TABLE refunds (charge_id TEXT, amount REAL);
    """
)
for c in json.load(open("/case/seed.json"))["charges"]:
    conn.execute(
        "INSERT INTO charges VALUES (:id, :email, :description, :amount, :created_at)", c
    )
conn.commit()
conn.close()

started = time.time()
try:
    result = {"outcome": "completed", "reply": handle_ticket(case["ticket"])}
except Exception as e:
    result = {"outcome": "failed", "error": repr(e)}
result["duration_s"] = round(time.time() - started, 2)

json.dump(result, open("/logs/agent/result.json", "w"), indent=2)
print(json.dumps(result))

Write the suite config and the agent

harbor.yaml is a Harbor job config. It refers to the API key as a ${VAR} placeholder that lmnr-cli fills in when the run starts, so the key itself does not end up in a file you commit:

evals/support/harbor.yaml

agents:
  - import_path: agent.agent:SupportAgent
    model_name: anthropic/claude-sonnet-5
    env:
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
datasets:
  - path: tasks
n_concurrent_trials: 4

The Harbor agent runs the driver inside the container. Everything it needs is already in the image, so install() has nothing to do. It passes self.model_name on to the app, which lets --model override the model for a run:

evals/support/agent/agent.py

from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext


class SupportAgent(BaseInstalledAgent):
    @staticmethod
    def name() -> str:
        return "support"

    async def install(self, environment: BaseEnvironment) -> None:
        pass  # everything the agent needs is already in the image

    @with_prompt_template
    async def run(
        self, instruction: str, environment: BaseEnvironment, context: AgentContext
    ) -> None:
        # "anthropic/claude-sonnet-5" -> "claude-sonnet-5"
        model = (self.model_name or "").split("/", 1)[-1]
        await self.exec_as_agent(
            environment,
            command="python /opt/harness/run_case.py",
            env={"AGENT_MODEL": model},
            timeout_sec=240,
        )

Add a case

The task points at the image by name. Harbor copies everything in the task’s environment/ folder into workdir (here /case) when the container starts:

evals/support/tasks/duplicate-charge/task.toml

[environment]
docker_image = "refund-agent-evals"
workdir = "/case"

[agent]
timeout_sec = 300

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

The [verifier.env] block gives the verifier the key it needs to call the LLM judge.

evals/support/tasks/duplicate-charge/instruction.md

Handle the support ticket in case.json.

evals/support/tasks/duplicate-charge/environment/case.json

{"ticket": "Hi, this is jacob@tracevell.com. I was charged twice for my October invoice, two identical charges of $49 on Oct 3. Can you refund one of them?"}

environment/seed.json: the database rows for this case

evals/support/tasks/duplicate-charge/environment/seed.json

{
  "charges": [
    {"id": "ch_8f3a1c", "email": "jacob@tracevell.com", "description": "Team plan, September", "amount": 49.0, "created_at": "2026-09-03T09:12:41Z"},
    {"id": "ch_2b7d90", "email": "jacob@tracevell.com", "description": "Team plan, October", "amount": 49.0, "created_at": "2026-10-03T09:12:44Z"},
    {"id": "ch_c41e66", "email": "jacob@tracevell.com", "description": "Team plan, October", "amount": 49.0, "created_at": "2026-10-03T09:12:45Z"}
  ]
}

The agent can read everything in environment/, so the reference answer must not go there. Put it in tests/, which Harbor only uploads when the verifier runs.

evals/support/tasks/duplicate-charge/tests/expected.json

{
  "refund_count": 1,
  "refund_amount": 49.0,
  "reply_should": "Confirm that one of the two October charges was refunded, and not promise anything beyond that."
}

The verifier first checks the database, which is the hard evidence of what the agent did. It then asks an LLM judge whether the reply is acceptable. It writes one reward per check, and each becomes its own score in Tracevell. If the agent run itself failed, both rewards are 0 and the judge is skipped:

evals/support/tasks/duplicate-charge/tests/test.sh

#!/bin/bash
python /tests/verify.py

tests/verify.py: the database check and the LLM judge

evals/support/tasks/duplicate-charge/tests/verify.py

import json
import sqlite3

from anthropic import Anthropic

expected = json.load(open("/tests/expected.json"))
result = json.load(open("/logs/agent/result.json"))

if result["outcome"] != "completed":
    print(f"agent run did not complete: {result}")
    json.dump({"refund": 0, "reply": 0}, open("/logs/verifier/reward.json", "w"))
    raise SystemExit(0)

reply = result["reply"]

# 1. Hard check: what actually happened in the database.
conn = sqlite3.connect("/srv/app/app.db")
count, total = conn.execute("SELECT COUNT(*), COALESCE(SUM(amount), 0) FROM refunds").fetchone()
refund_ok = count == expected["refund_count"] and total == expected["refund_amount"]
print(f"refunds: {count} totalling {total}")

# 2. LLM judge: is the reply to the customer any good?
verdict = Anthropic().messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=200,
    # Structured output, so the verdict is always valid JSON.
    output_config={
        "format": {
            "type": "json_schema",
            "schema": {
                "type": "object",
                "properties": {"pass": {"type": "boolean"}, "reason": {"type": "string"}},
                "required": ["pass", "reason"],
                "additionalProperties": False,
            },
        }
    },
    messages=[
        {
            "role": "user",
            "content": (
                f"A support agent wrote this reply to a customer:\n\n{reply}\n\n"
                f"The reply should: {expected['reply_should']}"
            ),
        }
    ],
)
judge = json.loads(verdict.content[0].text)
print(f"judge: {judge}")

json.dump(
    {"refund": int(refund_ok), "reply": int(judge["pass"])},
    open("/logs/verifier/reward.json", "w"),
)

Make test.sh executable with chmod +x.To add another case, copy the duplicate-charge directory, then change case.json and seed.json to describe a different ticket and account, and change expected.json to match. For example, a customer asking for a refund on a charge that should be declined expects "refund_count": 0.

Add secrets

Put ANTHROPIC_API_KEY in .env at the repo root, next to the LMNR_PROJECT_API_KEY that setup wrote. lmnr-cli fills every ${VAR} placeholder from your shell first, then from .env.local and .env. If one is missing, it stops before anything starts.Keep secrets and Harbor’s job output out of git:

.gitignore

.env
evals/*/jobs/

Run the evals

From the repo root:

npx lmnr-cli@latest eval run

lmnr-cli builds refund-agent-evals with your Docker, runs every task in every suite, and prints a link to the results. The support suite becomes an evaluation group in your project, and this run is a new evaluation in it.

Read the results

Open the evaluation. Each trial is a datapoint, with a refund score and a reply score. Because the app calls Laminar.initialize(), the trial’s trace shows the model calls and the list_charges / refund_charge tool calls as a transcript, so a 0 takes one click to explain.Locally, Harbor’s raw output is in evals/support/jobs/<timestamp>/. Each trial has agent/ (your result.json and logs) and verifier/ (reward.json and test-stdout.txt, which includes the judge’s reasoning).

Next steps

Run evals with lmnr-cli

Filter suites and tasks, run trials in Tracevell sandboxes, and wire it into CI.

The evals folder

Reference for lmnr.yaml, harbor.yaml, task.toml, and secrets.

Build a production harness

Grow this single module into a full stack with services and seeded data.

Read a failing trial

Open a low-scoring datapoint and read the app’s run as a transcript.

Overview

Tracing

Signals

Evaluations

Debugger

Datasets

Platform

Agent Evaluations Quickstart with lmnr-cli

Set up with your coding agent

Get started manually

Next steps

Run evals with lmnr-cli

The evals folder

Build a production harness

Read a failing trial

​Set up with your coding agent

​Get started manually

​Next steps

Run evals with lmnr-cli

The evals folder

Build a production harness

Read a failing trial

Set up with your coding agent

Get started manually

Next steps