Autonomous support agents grounded on private docs with human handoff.
STACK
Built autonomous support agents that answer from private documentation, escalate to humans when confidence drops, and learn from every resolved ticket. Instrumented with full tracing and evaluation harnesses.
Support tickets were growing faster than the support team could hire. Most tickets were genuinely answerable from existing documentation, but customers didn’t want to search docs themselves, and a naive chatbot that occasionally invented answers was worse than no chatbot at all — it eroded trust the first time it confidently gave a wrong answer.
Built an agent that only answers from retrieved, cited documentation — never from its own unsupported knowledge — and is instrumented to recognize its own uncertainty and hand off to a human rather than guess. Every resolved ticket feeds back into an evaluation set so accuracy is measured continuously, not just at launch.
The agent is architecturally prevented from answering without a citation — if retrieval doesn’t surface a confident source, it escalates instead of generating an unsupported answer.
A separate scoring pass estimates answer confidence before it reaches the customer; anything below threshold routes to a human with the agent’s partial reasoning attached, so the human isn’t starting from zero.
Every resolved ticket — agent-handled or human-escalated — becomes a labeled example in an evaluation set, so accuracy regressions from prompt or model changes are caught before they reach customers.
confidenceGate.ts
1async function answerOrEscalate(query: string) {2 const { docs, citations } = await retrieve(query)3 if (citations.length === 0) return escalate(query, 'no_grounding')45 const draft = await generateAnswer(query, docs)6 const confidence = await scoreConfidence(draft, docs)78 if (confidence < CONFIDENCE_THRESHOLD) {9 return escalate(query, 'low_confidence', { draft, confidence })10 }11 return { answer: draft, citations }12}
Built the citation-required retrieval pipeline against the existing documentation set.
Added the confidence gate and human handoff flow with reasoning trace attached.
Shipped the continuous evaluation harness and rolled out to the full ticket volume.
Forcing the agent to cite or escalate — never generate unsupported — did more for customer trust than any amount of prompt tuning aimed at "sounding confident."
A confidence score is only useful if it’s validated against real escalation outcomes over time, not just tuned once at launch and left alone.
If this project resonates with what you're building, let's talk. I take on a limited number of projects each quarter.
hello@martinsai.name.ng · Response within 24h