Metrics & Model Comparison

FleetQ tracks performance metrics across all experiments, agents, and LLM providers. Use metrics to compare model performance, optimize costs, and identify bottlenecks in your AI workflows.

What's tracked

Every AI run records the following dimensions:

Metric	Description
response_time	End-to-end latency in milliseconds
input_tokens	Prompt tokens consumed
output_tokens	Completion tokens generated
cost_credits	Cost in credits (1 credit = $0.001)
success_rate	Ratio of successful completions to total attempts
quality_score	Evaluator-assigned score (0–100)
throughput	Runs completed per hour per agent

Metric aggregation

Raw metrics are aggregated on two schedules via the metrics:aggregate Artisan command:

Hourly — runs at the top of every hour
Daily — runs at 01:00 UTC

Aggregations are stored in the metric_aggregations table and power dashboard charts without hitting raw event rows on every page load.

bash

# Trigger aggregation manually
php artisan metrics:aggregate --period=hourly
php artisan metrics:aggregate --period=daily

Model comparison

The /metrics/models page lets you compare LLM providers side-by-side across cost, latency, quality scores, and error rates. Use it to decide which model is best suited for a given task type.

Run the same experiment with two different providers, then use the model comparison view to compare quality vs. cost. Cheaper models often match expensive ones for structured extraction tasks.

bash

# Fetch model comparison data via API
curl https://fleetq.169.58.89.204.sslip.io/api/v1/dashboard \
  -H "Authorization: Bearer YOUR_TOKEN"

Revenue attribution

AttributeRevenueAction links experiment outcomes to business metrics, allowing you to track the ROI of AI workflows. Attach a revenue metric to any experiment to see which agents and models drive the most value.

Dashboard KPIs

The main dashboard surfaces key metrics at a glance:

Active experiments & success rate
Total spend (credits) this period
Agent utilization & queue depth
Top providers by cost and volume

The same data is available programmatically via the system_dashboard_kpis MCP tool, so your agents can reason about platform health.

Semantic cache performance

The AI gateway caches semantically similar prompts using pgvector (cosine similarity threshold 0.92). Monitor cache efficiency via MCP or the API to understand how much spend is being avoided.

MCP tools

semantic_cache_stats   // hit rate, savings, total cached entries
semantic_cache_purge   // flush cache entries (admin only)

The semantic cache is intentionally shared across teams (no tenant isolation) to maximise hit rates. No team data is exposed — only the cached model response is reused.