10 Metrics That Matter for AI Review Quality

Inter-Rater Reliability

When two reviewers independently evaluate the same output, how often do they agree? This is the foundation of every other quality metric. If reviewers disagree on pass/fail, your error rates and trend scores measure noise, not signal.

Formulas:

Percent Agreement = (paired reviews with same verdict) ÷ (total paired reviews) × 100%

For categorical decisions, compute Cohen's Kappa to adjust for chance agreement:

κ = (Po − Pe) ÷ (1 − Pe)

Example: You double-review 150 outputs monthly. Reviewers agree on pass/fail in 123 cases → 82% agreement. Cohen's Kappa = 0.74. The 27 disagreements cluster on "unsupported claims" — your criteria list the error type but lack concrete examples. Add two gold-standard outputs to the reviewer UI and rerun calibration.

Targets: Aim for 80%+ percent agreement and κ > 0.70 for well-defined tasks. Below 70% agreement, pause scaling review volume and fix criteria before adding headcount. Run a 30-minute calibration session before any major prompt or model change — see pre-ship verification for a calibration playbook.

Review Completion Rate

What percentage of assigned tasks get reviewed within the SLA? A low completion rate means reviewers are overwhelmed, tasks are stuck in queues, routing sends work to unavailable reviewers, or SLAs are unrealistic for task complexity.

Formula:

Completion Rate = (tasks reviewed within SLA) ÷ (tasks assigned) × 100%

Track per reviewer, per skill category, and per time window. A healthy aggregate can hide a backlog in one high-risk category.

Example: Last week: 3,400 tasks assigned, 3,058 completed within the 2-hour SLA → 89.9% completion rate. Drilling down: Tier 1 customer-facing at 97%, Tier 2 technical summaries at 71%. The Tier 2 gap maps to a single reviewer pool that lost two people — reroute or extend SLA for that category until staffing recovers.

Targets: Tier 1 (critical) should target 98%+ completion. Tier 2 might accept 90% with documented exceptions. Sustained completion below 85% anywhere is a capacity or routing problem, not a reviewer performance problem — fix the system before blaming individuals.

Time-to-Decision

How long from task submission to final review decision? This measures end-to-end review latency — queue time plus active review time. It is not just about speed. It tells you whether your review process adds acceptable delay to your AI pipeline.

Formula:

Time-to-Decision = timestamp(final_verdict) − timestamp(task_submitted)

Report P50 (median) and P95. Averages lie when a few tasks sit for hours. Decompose into queue wait vs active review to know whether to add reviewers or simplify criteria.

Example: Tuesday's production batch: P50 time-to-decision = 22 minutes, P95 = 3.1 hours. Queue wait accounts for 78% of P95 latency — reviewers are not slow; routing sends overnight volume to a US-only pool. Fix: follow-the-sun assignment or async reviewer coverage for off-hours spikes.

Targets: Real-time customer chat might require P95 <10 minutes. Batch report generation might allow P95 <4 hours. Document latency budgets alongside your SLAs so product and ops share expectations. If time-to-decision consistently exceeds SLA, you need more reviewers, better routing, or narrower criteria — not faster clicking.

Escalation Rate

What percentage of tasks require escalation to a senior reviewer or tiebreaker? Some escalation is healthy — it means first-line reviewers know their limits. Chronic high escalation means criteria are ambiguous, routing sends complex work to generalists, or reviewers lack confidence to decide.

Formula:

Escalation Rate = (tasks escalated) ÷ (first-line reviews completed) × 100%

Track escalation reasons, not just rate. "Unclear criteria" and "beyond my expertise" point to different fixes.

Example: 2,800 first-line reviews last month, 231 escalated → 8.3% escalation rate. Within range — but 58% of escalations cite "policy edge case" for refund letters. Route refund disputes directly to Tier 2 and publish a decision tree so first-line reviewers stop guessing.

Targets: 5–12% escalation is typical for mixed-complexity queues. Sustained rates above 15% indicate routing or training problems. Near-zero escalation may mean reviewers are guessing instead of escalating — cross-check agreement on borderline tasks.

False Detection Rate

How often do reviewers flag outputs as problematic when they are actually fine? False positives waste capacity, train reviewers to dismiss flags as noise, and erode trust in the review function. This metric is the mirror of your precision — are you catching real errors without crying wolf?

Formula:

False Detection Rate = (flagged outputs marked acceptable) ÷ (total flagged outputs) × 100%

Measure at two points: automated pre-screen flags and reviewer-initiated rejections. Each layer can be miscalibrated independently.

Example: Reviewers flagged 620 outputs for substantive issues last week. After secondary review or customer delivery, 186 were cleared without edits — acceptable as-is. False detection rate = 186 ÷ 620 = 30%. At 7 minutes per review, that is 21.7 hours of wasted capacity. Root cause: criteria treat "awkward phrasing" as blocking when it should be advisory.

Targets: Reviewer false detection below 15% for mature programs. Automated pre-screen FPR below 20%. If reviewers flag 30% of outputs but only 5% actually need changes, your detection criteria are too aggressive or poorly defined — tighten definitions before adding reviewers.

Reviewer Productivity

How many tasks does each reviewer complete per hour? This is not a metric to optimize blindly. Pushing productivity too hard degrades inter-rater reliability and inflates false detection. Use productivity to identify outliers, forecast staffing, and spot training gaps.

Formula:

Productivity = reviews completed ÷ reviewer active hours

Count only active review time — exclude calibration, meetings, and breaks unless you are measuring total capacity. Segment by task type; a single aggregate hides complexity differences.

Example: Team of six reviewers completes 2,160 reviews in a 40-hour week. Aggregate productivity = 2,160 ÷ (6 × 40) = 9 reviews/hour/reviewer. Outliers: two seniors at 15/hour on routine FAQ drafts (possible corner-cutting — spot-check their agreement), one junior at 5/hour on the same task type (training gap). Reassign accordingly.

Targets: Use productivity for capacity planning, not leaderboard rankings. If volume grows 25% next quarter and productivity holds at 9/hour, you need roughly two additional FTEs or routing improvements. Investigate anyone more than two standard deviations below team median on the same task types.

Cost Per Review

Total review spend divided by total reviews completed. Includes reviewer compensation, platform costs, tooling overhead, and allocated manager time. This metric drives automation decisions: if catching an error costs more than the error would cost, your review strategy needs redesign.

Formula:

Cost Per Review = (reviewer labor + platform fees + tooling overhead) ÷ (reviews completed)

Segment by task category. A stable aggregate can hide expensive senior review on tasks a junior could handle.

Example: Monthly spend: $36,000 reviewer labor + $2,400 platform + $600 tooling = $39,000. Reviews completed: 9,800. Cost per review = $39,000 ÷ 9,800 = $3.98. High-complexity legal summaries cost $9.20 each; low-risk internal drafts cost $1.60. Segmenting reveals where selective automation or routing changes have the highest ROI.

Targets: Cost per review should be stable or declining as your process matures. Spikes above 25% month-over-month warrant investigation — often a routing change sent simple tasks to expensive senior reviewers. Compare against expected error cost: if catching a $400 compliance failure costs $4 to review, the math works.

Quality Trend Score

A composite metric tracking whether AI output quality is improving or degrading over time, as measured by review outcomes. Unlike a snapshot error rate, the trend score tells you direction — critical after model migrations, prompt changes, or seasonal traffic shifts.

Formula:

Quality Trend Score = (outputs passing review without edits) ÷ (total outputs reviewed) × 100%

Compute over rolling 7-day and 30-day windows. Compare windows to detect acceleration — a flat 30-day score can hide a bad recent week.

Example: 30-day rolling pass-without-edits rate: 78.4%. Seven-day window: 71.2%. The gap signals recent degradation, not a long-term slide. Correlating with deployment logs shows a prompt v3.1 rollout three days ago — roll back or fix retrieval before the 30-day number catches up.

Targets: Define acceptable bands per task type. A declining trend score means either the AI model is getting worse, your quality standards are shifting, or reviewers are applying stricter interpretation — investigate which before changing production routing. Pair with model version tags in every audit record.

Coverage Percentage

What percentage of your AI outputs are actually being reviewed? If you review 500 out of 5,000 daily outputs, your 10% coverage leaves significant risk unexamined. Coverage should be a conscious decision, not an accident of capacity constraints.

Formulas:

Coverage = (outputs human-reviewed) ÷ (total outputs produced) × 100%

Risk-Weighted Coverage = Σ(reviewed × risk_weight) ÷ Σ(total × risk_weight)

Risk weights might be 3× for customer-facing, 2× for client deliverables, 1× for internal drafts.

Example: 42,000 outputs produced, 5,040 human-reviewed → 12% raw coverage. Risk-weighted: Tier 1 customer-facing at 96%, Tier 2 at 28%, Tier 3 internal at 3%. The low raw number is intentional — you are covering what matters. Document the uncovered 72% of Tier 2 as accepted risk with executive sign-off.

Targets: Tier 1 should approach 100% coverage. Tier 2 depends on error rate and trend score — expand coverage when Tier 2 error rate exceeds threshold. Revisit coverage quarterly as models improve. Know what you are not reviewing and why.

Customer Satisfaction Correlation

Does review quality correlate with downstream customer satisfaction? This is the hardest metric to compute and the most important. Internal dashboards can look healthy while customers still complain — if your error taxonomy does not match what users care about, you are optimizing the wrong things.

Formulas:

Δ CSAT = CSATreviewed − CSATunreviewed

Support Ticket Rate = tickets attributed to AI output ÷ (outputs delivered) × 1,000

Correlate reviewed vs unreviewed cohorts on the same output types. Control for channel and customer segment.

Example: Post-delivery surveys on AI-drafted support replies: reviewed outputs score CSAT 86%, unreviewed CSAT 73%. Support ticket rate: 0.7 per 1,000 reviewed sends vs 2.1 unreviewed. The 13-point CSAT gap justifies review spend on customer-facing sends. Internal draft summaries show no CSAT difference — deprioritize human review there and reallocate capacity.

Targets: Reviewed outputs should beat unreviewed by a meaningful margin on high-risk surfaces. If internal metrics improve but Δ CSAT is near zero, revisit your pass/fail criteria. Trust the customer signal when it diverges from reviewer verdicts — see when quality metrics mislead.

Putting These Metrics to Work

Do not try to track all ten on day one. Start with the trio that answers: Are reviewers calibrated? Is the pipeline keeping up? Are decisions fast enough?

1. Inter-rater reliability — confidence that every other metric means something
2. Review completion rate — whether assigned work actually gets done within SLA
3. Time-to-decision — whether review latency is acceptable for your product

Add cost per review and coverage percentage once you have four weeks of stable data. Layer in false detection rate, reviewer productivity, and escalation rate when optimizing routing and staffing. Add quality trend score and customer satisfaction correlation when you are ready to connect review operations to business outcomes.

Review the full stack weekly in a 30-minute ops sync. Red metrics get owners and deadlines; green metrics still get spot-checked — complacency is how drift accumulates. Publish a one-page summary for leadership: three headline numbers, two trends, one action item. Executives do not need ten charts; they need to know if review quality is improving and what you are doing about it.

Review quality is not a single dashboard widget. Inter-rater reliability without completion rate tells you reviewers agree — but not whether work ships on time. Low cost per review without customer satisfaction correlation tells you you are efficient — but not whether anyone notices. Track these ten together, and your review process becomes a feedback loop that gets measurably better — not a cost center that looks busy until a customer proves otherwise.

The goal is not a perfect dashboard. It is a review process that improves every quarter. Wire these metrics into alerts, revisit definitions when prompts change, and run calibration before every major launch. Start with three metrics, add five more as data matures, and close the loop with customer correlation when you are ready to prove ROI.

• Use the visual builder to configure quality gates and metric thresholds for your review pipeline.
• Open the sandbox to see how review results generate quality metrics.
• Reference the API reference for metrics and analytics endpoints.