Skip to main content
LLM evaluations are the offline testing layer for your prompts and LLM pipelines. Define inputs, run them through the version you want to test, score the outputs against target data, and compare across runs to see if anything regressed. To evaluate an agent end to end in a working copy of its environment, see Agent evaluations.

Introduction

What evaluations are and when to use them.

Quickstart

Write and run your first evaluation in five minutes.

Concepts

Datapoints, executors, evaluators, and how they map to traces.

Compare runs

Groups, the progression chart, and side-by-side comparison.

Datasets

Back your evaluation with a Tracevell dataset instead of hardcoded data.

Manual API

Lower-level control for pipelines where evaluate() is too opinionated.

Self-hosted

Point the SDK at a self-hosted Tracevell instance.
Next: turn failing traces into reusable evaluation data with Datasets, or catch regressions in production with Signals.