Why RAG Demos Lie to You
There's a gap between a RAG demo that impresses a CTO and a RAG system that survives contact with 10,000 daily users. The demo has three curated documents, a perfect query, and someone who already knows what the right answer looks like. Production has messy PDFs, users who ask things you never anticipated, and no one watching every response.
The Three Failure Modes We See Most
After 30+ production RAG deployments, the same failure patterns emerge: retrieval failures, context window abuse, and evaluation blindness.
- Retrieval failures: The right chunk exists in the corpus but the retriever never finds it — usually a chunking or embedding mismatch problem.
- Context window abuse: Stuffing 20 chunks into the prompt because 'more context is better' — it isn't. Models lose focus in long contexts.
- Evaluation blindness: Teams that ship without a systematic eval harness have no idea whether the next document they add will break existing answers.
The Evaluation Framework We Use Before Shipping
Before any RAG system goes to production, we run it through a four-stage evaluation gate. Each gate has a pass threshold — if it doesn't pass, it doesn't ship.
- Stage 1 — Retrieval precision: Given a ground-truth Q&A set, does the top-k retrieval include the correct chunk? Target: >85%.
- Stage 2 — Answer faithfulness: Is the generated answer grounded in the retrieved context? We use an LLM-as-judge pattern here.
- Stage 3 — Adversarial probing: Can a user cause the system to answer questions that are out of scope? If yes, guardrails need tightening.
- Stage 4 — Latency under load: p95 response time under simulated concurrent load. Most teams discover their vector DB isn't indexed properly here.
Chunking Strategy Matters More Than Model Choice
Teams spend weeks debating GPT-4o vs. Claude vs. Gemini. They spend hours on chunking strategy. This is backwards. We've seen the same model produce dramatically different results with different chunking approaches on the same corpus. Semantic chunking with overlap, tuned to the document type, consistently outperforms naive fixed-token chunking by 15–30% on retrieval precision.
What Good Looks Like in Production
A production RAG system that performs well has: a retrieval stage you can evaluate independently, a generation stage with explicit faithfulness checks, a feedback loop that surfaces low-confidence responses for human review, and monitoring dashboards that alert on retrieval degradation as the corpus grows. None of these are hard to build. Most teams just don't know to build them.
Key Takeaways
- Retrieval quality is the highest-leverage variable in RAG performance — optimize it first.
- Build an eval harness before you ship, not after you get complaints.
- Semantic chunking tuned to document type beats fixed-token chunking consistently.
- LLM-as-judge faithfulness scoring is the fastest way to catch hallucinations at scale.
- Monitor retrieval precision as your corpus grows — it degrades silently.