Can AI actually replace your support team? We tested it.
Last October, we ran an experiment that made some of our team uncomfortable. We routed a portion of our real customer support tickets through Relay — our own AI support product — without a human reviewing every response before it went out.
90 days. Real tickets. Real customers. Here's what happened.
The Setup
We didn't go in blind. We trained Relay on six months of past support conversations, our documentation, and a library of edge cases our team had flagged as tricky. We set confidence thresholds: anything below 85% certainty got flagged for human review before sending.
About 60% of incoming tickets went through Relay autonomously. The rest still touched a human.
What Worked
The obvious stuff worked immediately. Password resets, billing questions, plan comparisons, feature explanations — Relay handled these faster and more consistently than our human team. Response time dropped from an average of 4.2 hours to 11 minutes for those ticket types.
Customer satisfaction scores for those tickets were actually slightly higher than for human-handled tickets. Consistency matters. Relay never had a bad day, never sent a clipped response because it was tired.
By week six, we'd extended autonomous handling to 74% of tickets.
What Didn't Work
Emotionally charged tickets. This was the uncomfortable part.
When a customer was genuinely upset — frustrated with a billing error that had persisted for weeks, or dealing with data loss during a critical moment — Relay's technically correct responses landed badly. The words were right. The tone was off.
We also hit edge cases that exposed gaps in training. A customer asked about an integration we'd deprecated eight months ago. Relay confidently explained how to set it up. That integration no longer existed.
Human review caught it before it went out, but it was a reminder that confident is not the same as correct.
The Numbers After 90 Days
The results were hard to argue with:
- Ticket resolution time dropped 67% across all categories.
- Human support hours dropped 52%. Our team shifted from answering tickets to reviewing AI responses, handling escalations, and improving training data.
- Customer satisfaction held at 4.4 out of 5 — the same as pre-experiment. We expected a drop. It didn't come.
- One significant miss in 90 days: the deprecated integration case above. No customer was materially harmed, but it reminded us that the system needs ongoing maintenance, not a one-time setup.
What We Actually Concluded
AI doesn't replace a support team. It changes what a support team does. Our team spent less time on repetitive tickets and more time on genuinely complex problems — the ones that require judgment, context, and empathy that no model handles well yet.
The economics work. The customer experience holds. But you can't set it and forget it. Relay needs feeding: new documentation, updated edge cases, regular audits of low-confidence decisions.
The teams that will get the most from AI support aren't the ones who want to eliminate headcount. They're the ones who want their best support people working on the hardest problems.
That's what we're building toward.