Skip to main content

Evaluation — LLM-as-Judge Quality Gates

The Evaluation domain gives you a reproducible way to measure the quality of your workflows, agents, and crews. Store test cases in a dataset, run any workflow against them, and score the outputs with an LLM judge. Use it for regression testing before a deploy, for A/B comparing two prompts, or for catching quality drift over time.

Scenario: A team owns a "summarise support ticket" workflow. They capture 50 real tickets with expected summaries as a dataset. Before every change to the underlying agent, they re-run the evaluation and reject the change if the average judge score drops below 0.85.

Core concepts

EvaluationDataset

A collection of rows, each with an input payload and an expected output. Datasets are team-scoped and versioned — you can edit rows, add new examples, and freeze versions for reproducible runs.

Flow evaluation run

A single pass of a workflow against every row in a dataset. Each row produces an actual output; the judge model scores it against the expected output and stores per-row plus aggregate metrics.

LLM judge

Any configured LLM (default claude-haiku-4-5-20251001) can be used as a judge. Pass a custom scoring prompt with {expected} and {actual} placeholders, or use the built-in similarity rubric.

Regression gates

A dataset can carry a target score. Runs that fall below the target fail loudly — wire them into your CI pipeline or a pre-deploy approval to block quality drops from shipping.

The Agentic AI Flywheel

Beyond on-demand scoring, the Evaluation domain runs a self-improving loop: failures continuously feed the eval set, are catalogued by failure mode, and are watched for drift in production. These pieces are opt-in via config/evaluation.php.

Self-growing eval set

Every EvaluationCase carries a provenance (manual, failure_lesson, task_validation, thumbs_down, or drift). Real failures and negative feedback are curated into the dataset automatically, so the eval set grows from production rather than by hand.

Auto-eval at triage

When an experiment transitions to a failed state, AppendRegressionCaseOnFailureListener appends a deferred, non-gating regression case capturing the seed input and the named failure state — deterministic, no extra LLM call. Deferred cases are tracked but don't fail a run until promoted to active.

Error-mode catalog

The ErrorMode domain names and counts recurring failure modes and lets you assign a remediation lever to each, turning one-off failures into a tracked, prioritisable backlog.

Drift monitor

DriftSignal records computed observations — input-distribution shift, eval-score decay, thumbs-down rate, and latency/cost spikes — against a rolling baseline, flagging breached thresholds for review.

Production-eval monitor

RunProductionEvalMonitorAction samples live traffic and writes an EvaluationMonitorSnapshot — average score, pass rate, active/deferred counts, and sampled count — giving you a point-in-time read of quality in production over time.

Evaluation lifecycle

  1. Build a dataset — curate a handful of representative rows. Real production inputs work best.
  2. Pick a workflow — any published workflow can be scored. No code changes required.
  3. Choose a judge — default is claude-haiku-4-5-20251001, or supply your own judge prompt for domain-specific rubrics.
  4. Run — RunFlowEvaluationAction fans out one workflow execution per row, then scores each result.
  5. Review — inspect per-row scores in the Evaluation panel. Failing rows surface first so you can diagnose quickly.

MCP tools

Agents and the AI assistant can manage evaluations directly via the MCP server. Five tools cover the full lifecycle — dataset CRUD, running evaluations, and fetching results.

Tool Description
evaluation_dataset_manage Consolidated CRUD for generic evaluation datasets (list, get, create, update, delete, add_row, remove_row).
evaluation_run Run a generic evaluation over a dataset (non-workflow scenarios).
flow_evaluation_dataset_create Create a workflow-specific evaluation dataset with a name and optional description.
flow_evaluation_run_start Start a flow evaluation run against a dataset with an optional custom judge model and prompt.
flow_evaluation_results Retrieve per-row actual output + judge score + aggregate metrics for a run.
drift_signal_list List recorded drift signals (score decay, thumbs-down rate, input shift, cost spikes).
evaluation_monitor_snapshots List production-eval monitor snapshots for quality-over-time tracking.
regression_case_append Append a regression case (e.g. a captured failure) to the eval set.
Start a flow evaluation via MCP 
flow_evaluation_run_start({
  "dataset_id": "018f1a2b-...",
  "workflow_id": "018f1a2c-...",
  "judge_model": "claude-haiku-4-5-20251001",
  "judge_prompt": "Score how faithfully the actual summary captures the key points of the expected summary. 0.0–1.0. Expected:\n{expected}\n\nActual:\n{actual}"
})

Regression testing pattern

Wire flow evaluation into your deployment workflow:

  1. Maintain a golden dataset per critical workflow.
  2. After any agent, skill, or workflow change, trigger flow_evaluation_run_start via MCP or the assistant.
  3. Use the returned aggregate score as a hard gate (≥ target) before merging or deploying.
  4. If a regression is detected, flow_evaluation_results tells you exactly which rows failed and why.
Combine evaluation with the Workflow versioning system: keep the golden dataset pointed at a frozen workflow version, so you can compare a new candidate revision side-by-side with the known-good baseline.

Related concepts

  • Workflows — the object under test for flow evaluations.
  • Skills — carry their own built-in guardrail evaluators for per-output quality checks.
  • Metrics & Comparison — aggregate evaluation results across runs and models.
  • Test Suites — project-level regression test suites for agent outputs.