How We Built a Real-Time AI Review Pipeline

The Problem With Synchronous Review

The naive approach — submit an output, wait for a human to review it, return the result — works for low-volume use cases. But at scale, it breaks down. Reviewers need time to read, evaluate, and respond. A thorough review takes 2–5 minutes. If your pipeline blocks on that, you've built a very expensive rate limiter.

Worse, synchronous review destroys tail latency. Median response time might look acceptable when reviewers are idle, but p95 and p99 explode the moment queue depth grows. Enterprise SLAs are written against percentiles, not averages. A pipeline that blocks on human judgment fails those SLAs the first time traffic spikes — and traffic always spikes.

Our Architecture: Async-First With Webhooks

We designed the pipeline around three principles:

1. Never block the caller. Task submission returns immediately with a task ID. The caller continues processing.
2. Route intelligently. Not all tasks need the same reviewer. Skill-based routing matches task requirements to reviewer qualifications.
3. Deliver asynchronously. Results are pushed to the caller via webhooks, not pulled via polling.

These principles map to a control-plane architecture. Your application owns business logic and user delivery; our platform owns routing, reviewer assignment, consensus computation, and webhook integrity. That boundary lets you swap review vendors or scale reviewer pools without rewriting application code. Every stage emits structured events — submitted, routed, claimed, reviewed, delivered — so you can trace any task from API call to webhook receipt when something breaks at 2 AM.

Task Lifecycle

Every task follows a predictable path through the system:

• Submitted — The API validates the payload, assigns an ID, escrows payment, and places the task in the routing queue. Validation includes schema checks, skill requirement verification, and idempotency key deduplication. Duplicate submissions with the same key return the existing task ID — critical for safe retries on network failures.
• Routed — The router evaluates required skills, priority, and reviewer availability. High-priority tasks go to on-call reviewers. Standard tasks enter the general pool. Routing completes in under 800ms at p95.
• Claimed — A qualified reviewer picks up the task. The system starts a session timer and monitors activity. Heartbeats detect abandonment; if a reviewer goes idle for more than 90 seconds, the task returns to the queue.
• Reviewed — The reviewer submits their assessment. For consensus tasks, the system waits for the required number of votes. Blind review ensures reviewers cannot see each other's decisions until consensus is computed.
• Delivered — The final result is posted to the client's webhook endpoint with an HMAC signature for verification. Failed deliveries retry with exponential backoff for up to 24 hours.

Each transition writes an immutable audit record: timestamp, actor, state, and payload hash. When a customer asks "why was this output rejected six weeks ago?", you reconstruct the full chain — not guess from logs that rotated out.

The Routing Algorithm

Our router considers four factors when matching tasks to reviewers:

Score = skill_match × 0.4 + availability × 0.3 + reliability × 0.2 + speed × 0.1

Skill match is binary — you either have the certification or you don't. Availability is real-time: how many tasks the reviewer is currently handling. Reliability is their historical accuracy rate. Speed is their average review time relative to the task complexity.

We version routing rules as configuration, not hardcoded logic. When we launched medical transcription review, we temporarily tightened skill requirements and increased the reliability weight from 0.2 to 0.35. Error rates dropped 40% in the first week. When error rates stabilized, we reverted the weight — logged, reversible, testable. See our task routing guide for the full strategy matrix.

Handling Failures

Reviewers miss deadlines, give inconsistent ratings, or abandon tasks mid-review. Our failure handling:

• Timeouts — If a reviewer doesn't submit within the SLA window, the task is rerouted to a backup reviewer. Cascading timeouts mean the second reviewer starts before the first SLA fully expires — we pre-assign backups for express-tier tasks.
• Consensus divergence — If two reviewers disagree by more than a threshold, a third reviewer breaks the tie. The tie-breaker sees both prior verdicts but not reviewer identities, reducing anchoring bias.
• Abandonment — If a reviewer starts but doesn't finish, the task returns to the queue with no penalty to the client. Abandonment rates above 5% per reviewer trigger a quality review, not silent routing exclusion.
• Webhook failures — Dead-letter queues capture payloads that exhaust retries. Clients receive a daily digest of undelivered webhooks so integration issues surface before verdicts go stale.

Failure handling is where most review systems quietly degrade. A router that reroutes slowly turns a 47-second median into a 4-minute p95. We alert when reroute rate exceeds 8% in any 15-minute window — that threshold preceded every capacity incident we've had in production.

Performance Numbers

After six months in production:

• Median time from submission to first review: 47 seconds
• Median time to consensus (3-vote tasks): 2.1 minutes
• Task completion rate: 99.2%
• Webhook delivery success rate: 99.97%

These numbers hold across peak traffic because we scale reviewer assignment horizontally, not sequentially. Parallel routing takes the fastest reviewer response, not the slowest. For 3-vote consensus tasks, three reviewers work simultaneously; the system waits only for the third vote, not three sequential review windows.

Submission API latency stays flat regardless of queue depth. p95 response time for POST /v1/tasks is 180ms — routing, escrow, and queue placement happen asynchronously after the response returns. Clients that mistakenly poll for results add load without improving delivery speed; webhooks cut median time-to-verdict awareness from poll-interval jitter to sub-second notification.

Observability and SLA Dashboards

Real-time pipelines fail silently without per-stage metrics. We instrument five layers:

• Submission latency — API response time, independent of review duration
• Routing latency — Time from queue entry to reviewer assignment
• Review latency — Time from claim to verdict, segmented by skill and priority tier
• Consensus latency — Additional wait for multi-vote tasks
• Delivery latency — Time from final verdict to successful webhook receipt

When end-to-end latency spikes, stage-level breakdowns tell you whether to hire reviewers, fix routing rules, or debug webhook retries. Blended averages hide the problem — a healthy submission path masks a broken delivery path. We publish p95 alongside median for every stage because enterprise customers write SLAs against percentiles.

What We'd Do Differently

If we were rebuilding today, we'd invest earlier in reviewer quality signals. Early on, we treated all reviewers equally. Now we weight reviewer reliability heavily — a task reviewed by a 98% accuracy reviewer is fundamentally different from one reviewed at 75%.

We'd also build better anomaly detection from day one. Spotting patterns like one reviewer consistently approving everything, or a sudden spike in task abandonment, requires purpose-built monitoring that we bolted on later. Anomaly alerts now fire within five minutes of deviation; early versions took days to surface the same patterns through manual QA sampling.

We'd run shadow mode longer before enabling verified-path holds. Sending 500 real outputs through review without blocking delivery would have surfaced routing gaps in our skill taxonomy before customers hit them. Our pre-ship verification checklist formalizes that sequence today.

Real-time human review is not about making humans faster — it is about making the system never wait for them. Async submission, parallel assignment, and webhook delivery turn review from a bottleneck into a background control plane. Models generate at machine speed; routing and consensus decide what ships.

Try It Yourself

The sandbox uses the same pipeline as production. Submit a task, watch it route, and see the review results come back in real time. It's the fastest way to understand how the architecture works in practice. Wire a test webhook endpoint, submit a task with min_reviewers: 2, and measure submission latency separately from time-to-verdict — you should see the decoupling immediately.