10 Metrics Every AI Quality Team Should Track

Error Rate

The foundational metric: what percentage of AI outputs contain errors? Define "error" precisely for your domain — factual inaccuracies, tone violations, policy breaches, formatting issues, or safety failures. A vague definition makes error rate meaningless across reviewers and task types.

Formula:

Error Rate = (outputs with ≥1 error) ÷ (total outputs reviewed) × 100%

Count an output as erroneous if any reviewer flags a substantive issue — not cosmetic nits. Segment by task type, model version, and prompt template. Aggregate error rate hides the spikes that matter.

Example: Your support-drafting pipeline reviewed 2,000 outputs last week. Reviewers flagged 47 with factual errors, policy violations, or incorrect customer details. Error rate = 47 ÷ 2,000 = 2.35%. Broken down: Tier 1 refund letters at 0.8%, Tier 2 technical explanations at 5.1%. The aggregate looked fine; the breakdown revealed where to focus.

Targets: Set thresholds per risk tier. Tier 1 (customer-facing, regulated) might target <1%. Tier 3 (internal drafts) might accept 5–8% with selective review. Track week-over-week trend, not just the absolute number — a 0.5-point rise over four weeks is model drift even if you're still "under target."

Time-to-Review

How long does an output sit in the review queue before a human evaluates it? This measures your pipeline latency. High time-to-review means review capacity isn't keeping pace with output volume, routing sends tasks to unavailable reviewers, or priority rules are misconfigured.

Formula:

Time-to-Review = timestamp(review_started) − timestamp(output_created)

Report P50 (median) and P95 — averages lie when a few tasks sit for hours. Set SLA targets per task type and alert when P95 breaches them for 15+ minutes.

Example: Last Tuesday's batch: P50 time-to-review = 18 minutes, P95 = 2.4 hours. Investigation shows P95 spikes correlate with EU business hours when your US-only reviewer pool is offline. Fix: add follow-the-sun routing or async reviewer coverage for overnight queues.

Targets: Real-time customer chat might require P95 <5 minutes. Batch report generation might allow P95 <4 hours. Document latency budgets alongside your review SLAs so product and ops share the same expectations.

Reviewer Agreement

When two reviewers independently evaluate the same output, how often do they agree? Low agreement signals unclear guidelines, inconsistent training, or ambiguous task definitions. Without agreement data, you cannot trust any other accuracy metric — reviewers may be measuring different things.

Formulas:

Percent Agreement = (paired reviews with same verdict) ÷ (total paired reviews) × 100%

For categorical pass/fail decisions, also compute Cohen's Kappa to adjust for chance agreement:

κ = (Po − Pe) ÷ (1 − Pe), where P_o is observed agreement and P_e is expected agreement by chance.

Example: You double-review 200 outputs monthly. Reviewers agree on pass/fail in 172 cases → 86% agreement. Cohen's Kappa = 0.78 (substantial agreement). The 28 disagreements cluster on "tone too casual" — criteria need a concrete rubric with examples, not adjectives.

Targets: Aim for 85%+ percent agreement and κ > 0.70 for well-defined tasks. Below 70% agreement, pause scaling review volume and fix criteria before adding headcount. Run calibration sessions with gold-standard examples until agreement recovers.

False Positive Rate

What percentage of outputs flagged for review were actually fine? A high false positive rate means your pre-review screening is too aggressive, wasting reviewer time on non-issues and training reviewers to dismiss flags as noise.

Formula:

False Positive Rate = (flagged outputs marked acceptable) ÷ (total flagged outputs) × 100%

Measure at two points: automated pre-screen flags and reviewer-initiated flags. Each layer can be miscalibrated independently.

Example: Your confidence-based auto-flagger routed 480 outputs to mandatory review last week. Reviewers cleared 137 without edits — acceptable as-is. False positive rate = 137 ÷ 480 = 28.5%. At 6 minutes per review, that's 13.7 hours of wasted capacity. Raising the confidence threshold from 0.72 to 0.78 drops FPR to 14% while catching 96% of true errors in shadow testing.

Targets: Automated pre-screen FPR below 20% for most domains. Reviewer FPR below 15% — higher rates suggest criteria are too strict or reviewers lack authority to approve borderline cases. Tune thresholds using shadow mode before changing production routing.

Cost-per-Review

The total cost of reviewing one output: reviewer time, tooling overhead, and opportunity cost of latency. This metric drives your automation decisions. If cost-per-review exceeds the value of catching that error, you need better automation, more selective review triggers, or a different model tier.

Formula:

Cost-per-Review = (reviewer labor + platform fees + tooling overhead) ÷ (reviews completed)

Include fully loaded labor cost — not just hourly wage. Add platform per-task fees, QA tooling, and manager overhead allocated per review.

Example: Monthly review spend: $48,000 reviewer labor + $3,200 platform + $800 tooling = $52,000. Reviews completed: 12,400. Cost-per-review = $52,000 ÷ 12,400 = $4.19. High-complexity legal summaries cost $11.40 each; low-risk FAQ drafts cost $1.85. Segmenting reveals where automation ROI is highest.

Targets: Cost-per-review should be stable or declining as process matures. Spikes above 25% month-over-month warrant investigation — often a routing change sent simple tasks to expensive senior reviewers. Compare cost-per-review against expected error cost: if catching a $500 compliance error costs $4 to review, the math works.

Throughput

How many outputs does your review team process per hour? Track per reviewer, per task type, and in aggregate. Throughput reveals capacity constraints, training gaps, and the impact of task complexity on review speed.

Formula:

Throughput = reviews completed ÷ reviewer active hours

Count only active review time — exclude breaks, meetings, and calibration sessions unless you're measuring total capacity. Use rolling 7-day windows to smooth daily variance.

Example: Team of eight reviewers completes 3,840 reviews in a 40-hour work week. Aggregate throughput = 3,840 ÷ (8 × 40) = 12 reviews/hour/reviewer. Outlier analysis: two senior reviewers at 18/hour on routine tasks (possible corner-cutting), one junior at 6/hour on the same task type (training gap). Reassign task types accordingly.

Targets: Do not optimize throughput blindly — pushing speed degrades agreement. Use throughput to forecast staffing: if volume grows 30% next quarter and throughput holds at 12/hour, you need roughly 2.4 additional FTEs or routing improvements. Investigate reviewers more than two standard deviations below team median.

Escalation Rate

What percentage of tasks get escalated to a second reviewer or domain expert? Some escalation is healthy — it means first-level reviewers know their limits. Chronic high escalation suggests routing rules send complex work to generalists, or criteria are ambiguous enough that nobody trusts first-pass decisions.

Formula:

Escalation Rate = (tasks escalated) ÷ (first-line reviews completed) × 100%

Track escalation reasons, not just rate. "Unclear criteria" and "beyond my expertise" point to different fixes.

Example: 4,200 first-line reviews last month, 344 escalated → 8.2% escalation rate. Healthy range. But 61% of escalations cite "policy edge case" for billing disputes — signal to route billing disputes directly to Tier 2 and update the policy decision tree.

Targets: 5–12% escalation is typical for mixed-complexity queues. Sustained rates above 15% indicate routing or training problems. Near-zero escalation may mean reviewers are guessing instead of escalating — check agreement on escalated vs non-escalated tasks.

Customer Satisfaction

The metric that actually matters: are end users satisfied with AI output quality? Internal error rates can improve while customer experience degrades — if you're measuring the wrong errors, or fixing outputs reviewers approve but customers reject.

Formulas:

CSAT = (satisfied responses) ÷ (total survey responses) × 100%

Quality NPS = % promoters − % detractors on AI-specific satisfaction surveys

Correlate reviewed vs unreviewed outputs: Δ CSAT = CSATreviewed − CSATunreviewed

Example: Post-delivery surveys on AI-drafted emails: reviewed outputs score CSAT 84%, unreviewed CSAT 71%. Support ticket rate on reviewed content is 0.9 per 1,000 sends vs 2.4 unreviewed. The 13-point CSAT gap justifies review spend — but only on customer-facing sends. Internal drafts show no CSAT difference; deprioritize review there.

Targets: Reviewed outputs should beat unreviewed by a meaningful margin on high-risk surfaces. If internal metrics improve but Δ CSAT is near zero, your error taxonomy may not match what customers care about. Trust the customer signal when it diverges from internal dashboards — see our analysis of when quality metrics mislead.

Model Drift

Is your model's performance changing over time? Monitor error rates by week and month, segmented by output type and prompt version. Model drift is gradual and easy to miss without consistent tracking — especially in language models, where degradation often manifests as subtle quality loss, not catastrophic failures.

Formula:

Drift % = (error_ratecurrent − error_ratebaseline) ÷ error_ratebaseline × 100%

Set baseline from the first stable four weeks after a model or prompt launch. Alert when drift exceeds 25% for two consecutive weeks or error rate crosses an absolute threshold.

Example: Baseline error rate after March prompt v2.3 launch: 1.8%. By mid-May, same task type at 3.1% with no intentional changes. Drift = (3.1 − 1.8) ÷ 1.8 = 72% relative increase. Root cause: upstream retrieval index stale — model answers confidently from outdated documents. Fix the index, not the model weights.

Targets: Weekly drift monitoring for production models. Monthly comparison across prompt versions. After any model migration, run parallel shadow evaluation for two weeks before declaring success — see handling model migrations for a full playbook.

Review Coverage

What percentage of AI outputs are reviewed by humans? This isn't "higher is better" — 100% coverage is usually wasteful. The goal is to review outputs that matter most: high-risk, high-visibility, or low-confidence content. Track coverage alongside error rate to find the sweet spot.

Formulas:

Review Coverage = (outputs human-reviewed) ÷ (total outputs produced) × 100%

Risk-Weighted Coverage = Σ(reviewed × risk_weight) ÷ Σ(total × risk_weight)

Risk weights might be 3× for customer-facing, 2× for internal client deliverables, 1× for internal drafts.

Example: 50,000 outputs produced, 6,200 human-reviewed → 12.4% raw coverage. Risk-weighted: Tier 1 customer-facing at 94% coverage, Tier 2 at 31%, Tier 3 internal at 2%. The low raw number is intentional — you're covering what matters. Uncovered Tier 2 at 69% is the conscious risk budget; document why.

Targets: Tier 1 should approach 100% coverage. Tier 2 depends on error rate — if Tier 2 error rate is 4% and uncovered volume is high, expand coverage or improve the model. Revisit coverage quarterly as models improve and risk tolerance shifts.

Building Your Metrics Stack

Don't instrument all ten metrics on day one. Start with the trio that answers: Are outputs good? Can we trust reviewers? Is the pipeline keeping up?

1. Error rate — baseline quality signal, segmented by task type
2. Time-to-review — pipeline health and SLA compliance
3. Reviewer agreement — confidence that error rate means something

Add cost-per-review and review coverage once you have four weeks of stable data. Layer in false positive rate, throughput, and escalation rate when optimizing routing and staffing. Add customer satisfaction and model drift when you're ready to connect internal metrics to business outcomes and long-term stability.

Review the full stack weekly in a 30-minute ops sync. Red metrics get owners and deadlines; green metrics still get spot-checked — complacency is how drift accumulates. Publish a one-page summary for leadership: three headline numbers, two trends, one action item. Executives don't need ten charts; they need to know if quality is getting better and what you're doing about it.

No single metric tells the whole story. Error rate without cost-per-review is incomplete. Throughput without reviewer agreement is meaningless. Customer satisfaction without review coverage hides whether your quality investment reaches the outputs customers actually see. Track these ten together, and you'll have the full picture — not a dashboard that looks good until a customer proves otherwise.

Start with error rate, time-to-review, and reviewer agreement. Those three give you a solid foundation. Add the others as your quality operation matures. Wire them into a dashboard, set alert thresholds, and revisit definitions quarterly — stale criteria produce stale metrics.

• Use the visual builder to configure quality gates and routing rules that feed your metrics pipeline.
• Open the sandbox to submit sample tasks and see how review outcomes map to quality KPIs.
• Reference the API reference for webhook payloads and analytics endpoints.