Skip to main content
lmnr-cli eval run runs every agent evaluation suite in your evals/ folder, locally on Docker or in Tracevell’s cloud sandboxes with --remote, and records each trial in Tracevell with its trace. Run it from the repo root, since that is where it looks for .env and .env.local. Use local mode while you iterate, after every change. Use --remote to run many more trials at once. lmnr-cli uploads the images, starts the run in Tracevell, and returns. Results appear in the project as trials finish.
Both modes need Docker on the machine where you run lmnr-cli, because that is where the images are built. In remote mode the built images are then uploaded.

​
In CI

Use --json and the exit code. A local run prints {mode, suites, failed, url}; a --remote run prints {mode: "remote", uploadId, sandboxId, suites, url} once the run has started. Set LMNR_PROJECT_API_KEY and the secrets your harbor.yaml references as CI secrets.

​
Remote limits and image requirements

  • One container per trial. docker-compose tasks don’t run remotely. cpus and memory_mb above the sandbox limit are clamped, and a run that exceeds the time limit is stopped.
  • The folder upload is capped at 1 GiB. Symlinks, .git, node_modules, virtualenvs, caches, and jobs/ directories are skipped, and .env* files are never uploaded. Keep large fixtures out of evals/.
  • Images need tar, gzip, and bash, plus su if a task runs as a non-root user. They’re built for linux/amd64 (emulated on Apple silicon), so architecture-specific downloads should use ARG TARGETARCH.
  • Concurrency is capped per project (50 sandboxes by default). A higher --n-concurrent queues rather than running more at once.
  • Large images may need a higher build_timeout_sec under [environment] in task.toml (default 600 seconds), which bounds image preparation plus sandbox start.
  • Self-hosted Tracevell: set LMNR_GRPC_PORT in your shell or .env for --remote runs, or the trial traces don’t arrive.

​
Next steps

The evals folder

Reference for lmnr.yaml, harbor.yaml, task.toml, and secrets.

Compare runs

Every run of a suite is charted in its group. Diff two runs side by side to see which cases regressed.

Read a failing trial

Open a low-scoring datapoint and read the app’s run as a transcript.

Troubleshooting

Fixes for stale images, missing env vars, and empty traces.