Designing a RAG Pipeline That Actually Works in Production
Voltra wanted to add AI-powered support to their B2B platform, but needed accuracy guarantees that typical chatbot solutions couldn't provide.
auth containing your JWT:{ "event": "auth", "token": "Bearer <JWT>" }65%
Ticket Deflection
Support tickets resolved without human intervention
1.4s
Response Time
Average time to first token in production
97%
Accuracy
Factual accuracy rate in generated responses
+18pts
CSAT
Improvement in customer satisfaction score
The Business Challenge
The knowledge base contained 200K+ documents across support articles, API docs, and internal runbooks. Off-the-shelf RAG solutions produced hallucinations at an unacceptable rate (23% of responses contained factual errors). The system needed to maintain sub-2-second response times while being honest about uncertainty.
Our Engineering Approach
We built a multi-stage retrieval pipeline with hybrid search (dense + sparse vectors), re-ranking, and a confidence scoring system that routes low-confidence queries to human agents. We also implemented a feedback loop that uses agent corrections to improve retrieval quality over time.
Key Architecture Decisions & Tradeoffs
Hybrid search over pure vector similarity
Pure vector search missed keyword-specific queries (error codes, API endpoints). By combining dense embeddings with BM25 sparse retrieval and reciprocal rank fusion, we improved retrieval relevance by 34%.
Confidence-based routing over always-generate
Rather than forcing the LLM to always produce an answer, we score retrieval confidence and route uncertain queries to human agents with relevant context pre-loaded. This dropped the hallucination rate from 23% to under 3%.
Streaming responses with citation chunking
We stream LLM responses token-by-token but inject citation markers in real-time, linking each claim to its source document. Users can verify any statement with one click.
“They didn't just build what we asked for - they challenged our assumptions and shipped something better.”
Marcus Rivera
VP of Product, Voltra