Skip to main content

Test Suites

A Test Suite is a reusable regression check for a project's agent outputs. It bundles a set of assertion rules and a quality threshold, then runs them against experiment results to produce a pass/fail Test Run with a score. Use suites to catch quality regressions before a change ships — the testing equivalent of unit tests for your agents.

Scenario: A project generates weekly market summaries. The team attaches a test suite with a 0.85 quality threshold and assertion rules ("must cite at least two sources", "no broken links"). Each new run is scored automatically; runs below threshold are flagged Failed and surface for review.

Core concepts

TestSuite

Team-scoped, optionally bound to a project. Carries a test_strategy, assertion_rules, a quality_threshold, and a rolling pass_rate.

TestRun

A single execution of a suite against an experiment. Records a status, a numeric score, structured results, optional agent feedback, and a duration in milliseconds.

Test strategy

One of full, lint_only, smoke, or regression — controls how thoroughly outputs are checked.

Quality threshold

The minimum score a run must reach to pass. Runs below it are marked failed so a quality drop can block a release.

Run statuses

A run moves pending → running → one of the terminal states: passed, failed, or skipped.

How runs are produced

  • RunRegressionTestsAction executes a suite's regression strategy against an experiment and records a TestRun.
  • EvaluateOutputAction scores an output against the suite's assertion rules and quality threshold.

In the UI

  • Test Suites lists every suite with its strategy, threshold, last-run time, and pass rate.
  • The suite detail page shows its assertion rules and the history of test runs with per-run scores and results.

MCP tools

Tool Description
test_suite_list List the team's test suites.
test_suite_get Fetch a suite with its rules and recent runs.
test_run Run a suite and return the resulting TestRun.
lint_run Run the lint-only strategy for a fast structural check.
Suites focus on a project's regression coverage. For prompt-level, LLM-as-judge scoring of workflow outputs against expected answers, pair this with the Evaluation domain.

Related concepts

  • Evaluation — LLM-as-judge scoring and golden datasets.
  • Projects — the container a suite is usually bound to.
  • Experiments — the run a suite evaluates.