What Production AI Review Actually Looks Like

The request path

Your app POSTs a task with the AI output, a JSON schema for the review UI, and a webhook URL. The platform normalizes the payload, validates the schema, assigns a priority tier, and routes to an available reviewer. For express tasks, the SLA clock starts immediately — typically 5 minutes for standard review, faster for premium tiers.

In practice, the submission looks like a structured API call, not a Slack message. A fintech team routing wire-transfer summaries might POST something like this:

• output — the model completion, often with structured fields extracted server-side
• schema — reviewer form definition: checkboxes for compliance clauses, free-text for corrections
• priority — critical, high, standard, or batch
• skills — tags like hipaa, spanish, or audio-qc that constrain routing
• webhook_url — HTTPS endpoint that receives the signed verdict

The API returns a task ID in under 200ms. Your application continues — it does not block on human judgment. That async contract is the foundation of every production pipeline we see scale past a few hundred tasks per day.

Queue design: how work actually flows

Production queues are not a single FIFO list. Mature pipelines run parallel queues segmented by priority, skill, and geography. A healthcare documentation team might operate four lanes:

• Critical lane — patient-facing discharge summaries. Dedicated on-call reviewers, 15-minute SLA, auto-escalation at 50% elapsed time.
• Express lane — same-day internal notes. General certified reviewers, 5-minute SLA, overflow to backup pool.
• Standard lane — batch chart abstractions. 4-hour SLA, pull-based assignment.
• Supervisor lane — consensus splits and policy edge cases. Senior reviewers only, no hard timeout — but alert if idle >30 minutes.

Queue depth is the early warning signal most teams ignore until customers complain. When express queue depth exceeds 20 tasks, P95 latency climbs even if individual reviewers are fast — because new tasks wait behind a backlog. Production operators watch depth per lane, not aggregate task count.

Routing uses metadata, not manual triage. A router scores eligible reviewers on skill match, current load, historical accuracy, and average speed. Tasks that require fda-21cfr certification never land on a general reviewer, even during a surge. That constraint prevents the fastest path from becoming the riskiest one.

SLA tiers you will actually ship

SLAs in production are contracts with your downstream systems. Your app, your support team, and your customers all assume a verdict arrives within a window. Here is a tier structure we see work across fintech, healthcare, and content moderation teams:

• Critical (P0) — 15-minute end-to-end. Regulatory, safety, or revenue-blocking outputs. On-call reviewers, automatic escalation, consensus optional but supervisor notified on any rejection.
• Express (P1) — 5-minute end-to-end. Customer-facing but not life-critical. Backup routing at 60% SLA elapsed, DLQ alert at breach.
• Standard (P2) — 4-hour end-to-end. Internal tools, drafts, async workflows. Pull queue, no on-call requirement.
• Batch (P3) — 24-hour end-to-end. Overnight processing, analytics exports. Light review or spot-check sampling.

SLA measurement includes queue time. A reviewer who completes in 90 seconds still breaches SLA if the task sat unclaimed for six minutes. Production dashboards split time-to-claim from time-in-review so you know whether to hire reviewers or fix routing.

What reviewers actually see

Reviewers don't get a chat window with the raw prompt. They get a structured form: the AI output on one side, validation fields on the other. For audio tasks, they scrub waveforms. For medical notes, they flag specific clauses. The UI is generated from your schema — which means bad schemas produce bad review experiences. Invest in schema design as much as model quality.

Consider three production UIs we see repeatedly:

• Clause-level medical review — each diagnosis statement is a toggleable row. Reviewers mark hallucinated ICD codes, missing disclaimers, or dosage errors without rewriting the full note. Corrections are field-scoped, which keeps audit diffs readable.
• Audio QC with waveform markers — transcribed segments sync to timestamps. Reviewers click a segment to flag misheard proper nouns or dropped negations. The webhook returns timestamp-indexed corrections your player can apply.
• Policy moderation for fintech — a checklist of 12 compliance statements with pass/fail per clause. One failed clause blocks approval. Reviewers can add a mandatory override reason that flows to the supervisor queue.

Schema mistakes show up in week one: free-text fields where enums belong, 40-field forms that reviewers skip, missing context panels that force tab-switching. The best production teams iterate schemas weekly using reviewer abandonment rate as the signal.

Consensus in practice

High-stakes tasks can route to multiple reviewers simultaneously. If all three agree, the webhook fires immediately. If two approve and one rejects, the task escalates to a supervisor queue. This isn't theoretical — it's how regulated teams avoid single-point-of-failure in human judgment.

Production consensus has explicit rules, not vibes:

• Quorum — typically 2-of-3 or 3-of-3 for Tier 1; 1-of-1 for Tier 2 with spot audits
• Agreement threshold — unanimous for medical; majority for content moderation with supervisor tie-break
• Parallel vs. sequential — parallel for speed (all reviewers see the task at once); sequential when later reviewers should be blind to earlier votes
• Escalation payload — supervisor sees all three verdicts, criterion scores, and reviewer notes — not just "disagreement"

A health-tech customer running 3-vote consensus on discharge summaries sees median consensus time of 2.1 minutes when reviewers work in parallel. Sequential consensus on the same volume averages 6.4 minutes — acceptable for audit-sensitive workflows, costly for express lanes.

Webhook delivery is the product

The review UI is visible; webhook delivery is invisible until it breaks. Production pipelines need:

• HMAC-signed payloads so clients can verify authenticity
• Exponential backoff retries (1m, 5m, 15m, 1h…)
• A dead-letter queue for permanently failed deliveries
• Idempotency so clients can safely retry processing

If your webhook endpoint returns 503 during a deploy, the review still happened — but your app never heard about it. DLQs and retry logs are how you close that gap.

Payload design matters as much as retry logic. Production webhooks should carry: task_id, verdict, corrected_output (if any), reviewer_ids, criterion_scores, consensus_metadata, completed_at, and an idempotency_key. Your downstream service stores that tuple immutably — six months later, when a regulator asks why a specific output shipped, you reconstruct the decision from the webhook archive, not Slack threads.

Metrics that matter

Vanity metrics: total tasks reviewed. Useful metrics:

• First-pass approval rate — how often AI outputs pass without correction. A healthy range depends on domain; 70–85% for generative content, 90%+ for extraction tasks. A sudden drop signals model or prompt drift.
• Consensus agreement rate — reviewer alignment on multi-vote tasks. Below 85% means your criteria are ambiguous or your training is stale.
• P50/P95 review latency — split by queue lane. Express P95 above 8 minutes means capacity or routing failure, not slow reviewers.
• Time-to-claim vs. time-in-review — tells you whether to add reviewers or simplify schemas.
• Webhook delivery success rate — are clients actually receiving results? Alert below 99.5%.
• Reviewer abandonment rate — tasks claimed but not completed. Above 3% usually indicates bad UX or unclear criteria.

Track these weekly. A dropping first-pass rate means your model drifted. Rising P95 latency means reviewer capacity is tight. Rising DLQ depth means your integration layer is failing — a different problem requiring a different fix.

Three production snapshots

Abstract architecture is easier to grasp with concrete volumes:

• Mid-size health documentation (8,000 tasks/day) — 60% standard lane, 30% express, 10% critical. Three-vote consensus on critical only. Peak hours 9–11 AM EST; on-call reviewers in US and Philippines for follow-the-sun coverage. Weekly SLA compliance: 97.8% express, 99.4% critical.
• B2B content generation (2,400 tasks/day) — 100% express lane, single reviewer with 10% spot-check audit by a second reviewer. First-pass approval rate tracked per customer workspace — one customer's rate dropped from 82% to 61% after a prompt change, caught before their end users noticed.
• Call-center QA audio (15,000 tasks/day) — batch lane overnight, express lane for escalated calls. Waveform UI with segment-level flags. Webhook payloads include timestamp arrays so the CRM can jump to flagged audio. Biggest week-one issue: 12 MB audio files timing out on upload — solved with pre-signed S3 URLs in the task payload.

What breaks in week one

Common production surprises: oversized payloads timing out, reviewers skipping required fields, webhook endpoints not handling duplicate deliveries, and timezone bugs in SLA calculations. Plan for all four before launch — they're predictable.

Add these to your pre-launch checklist, learned from teams who did not:

• Payload limits — cap inline content at 1 MB; reference large assets via URL. Validate on submission, not at routing time.
• Required field enforcement — block submit until mandatory criteria are scored. Soft validation invites skipped checks under time pressure.
• Duplicate webhook handling — store idempotency_key in a dedup table with 72-hour TTL. Retries are a feature, not a bug.
• SLA timezone — define whether SLA clocks use UTC, reviewer local time, or customer contract timezone. Document it in the API spec.
• Consensus thundering herd — when three reviewers finish within seconds, three near-simultaneous webhooks can race. Emit one consolidated webhook after quorum, not one per vote.

Production AI review is an integration problem dressed up as a quality problem. The model generates; the pipeline decides when a human weighs in, how fast they must respond, and how reliably your app learns the outcome. Teams that instrument queue depth, SLA splits, and webhook health treat review as infrastructure — and ship AI products that survive their first traffic spike.

Your first-month operating rhythm

After launch, run this cadence until metrics stabilize:

1. Daily: Check express queue depth, webhook DLQ size, and any SLA breaches from the prior 24 hours.
2. Weekly: Review first-pass approval rate by task type, consensus agreement rate, and top reviewer abandonment reasons.
3. Biweekly: Schema iteration session with lead reviewers — cut fields that are never used, add fields that appear in override notes.
4. Monthly: SLA tier review against actual P95 data. Adjust capacity, backup pools, or tier definitions — not just reviewer training.

By day 30 you should know your bottleneck: model quality, reviewer capacity, schema UX, or webhook reliability. Each has a different fix. Guessing which one costs you another month of customer-facing errors.