Retrieval-Augmented Generation is the industry standard for grounding language models in proprietary data. It dramatically reduces hallucinations and enables domain-specific expertise that a general-purpose model simply cannot provide.
Getting RAG working in a prototype takes an afternoon. Getting it to work reliably in production, with real users, real documents, and real stakes attached to every answer, is a different problem entirely.
The difference comes down to three pipeline stages, each of which has a prototype-level approach and a production-level approach. Here is what the production level actually looks like.
Stage 1: Pre-Retrieval, Indexing and Data Quality
Most RAG failures originate before retrieval happens at all. The quality of what you get back is determined entirely by the quality of what you put in. In production, the basic approach of splitting documents into fixed-size chunks, embedding them, and storing them is not sufficient.
Chunking Strategy
Fixed-size text splitting is fast to implement and poor in production. It splits on character count, which means a single coherent idea frequently gets cut across two chunks. The model retrieves half the context it needs and either hallucinates the rest or produces an incomplete answer.
Production systems use one of two better approaches.
Semantic chunking splits on topic change rather than character count. The chunker detects when the subject matter shifts and starts a new chunk there. The result is chunks that contain complete, contextually coherent ideas, exactly what retrieval needs.
Structure-aware splitting respects the document's own organisation: paragraphs, headings, table boundaries, section breaks. This works particularly well for structured documents like contracts, policies, and technical manuals where the document structure itself encodes meaning.
Metadata Enrichment
Raw text chunks are only part of what gets stored. In production, every chunk should carry metadata: source document name, creation or modification date, document type, and, critically for enterprise deployments, user permissions or department tags.
Metadata-filtered retrieval is non-negotiable for enterprise security. Without it, a query from one department can surface confidential data from another. With it, the retrieval layer only returns chunks the requesting user is authorised to see.
This is not an optimisation. It is a governance requirement for any system touching sensitive operational data.
Hierarchical Indexing
For large knowledge bases, thousands of documents and millions of chunks, flat indexing becomes a performance and relevance problem simultaneously. A hierarchical index solves both.
The top level stores document-level summaries. The lower level stores detailed chunks. A query first hits the summary index to identify the most relevant documents, then retrieves detailed chunks only from those documents. Retrieval is faster and the signal-to-noise ratio is significantly higher.
Stage 2: Retrieval, Precision Over Volume
Retrieving the top-K most similar chunks by cosine distance is the starting point. In production, it is rarely sufficient on its own.
Query Transformation
A user's conversational query is often a poor search query. Follow-up questions assume context that the retrieval layer has no access to. Ambiguous phrasing produces inconsistent results.
Production systems transform the query before retrieval using two techniques.
Query rewriting passes the user's query through a fast LLM call that rewrites it into a standalone, search-optimised form. The follow-up question becomes a complete, unambiguous query.
Multi-query retrieval generates two or three alternative phrasings of the same question and runs all of them in parallel. The results are merged and deduplicated. This casts a wider net and consistently surfaces relevant chunks that a single query formulation would miss.
Hybrid Search
Dense vector search retrieves semantically similar content. It works well for conceptual questions but struggles with exact terms: product codes, carrier names, specific contract clauses, regulatory references.
Sparse keyword search (BM25) handles exact terms well but misses semantic similarity.
Production RAG systems use both in combination. The outputs are merged using a reciprocal rank fusion algorithm that balances semantic and exact-match relevance. Neither approach alone is as reliable as the two together.
Reranking
Initial retrieval is optimised for speed. It returns a broad candidate set quickly. Reranking is the precision pass.
After retrieving the top 20 or so candidates, a cross-encoder model, smaller and slower than the main LLM but trained specifically for relevance scoring, evaluates each candidate against the original query and produces a ranked list. The bottom half of that list is discarded. Only the genuinely relevant chunks reach the final generation step.
This prevents the context window from being filled with marginally relevant content that dilutes the quality of the response.
Stage 3: Post-Retrieval and Generation, Context and Control
Retrieval produces a set of relevant chunks. The post-retrieval stage determines what the model actually sees and what the user ultimately receives.
Contextual Compression
Retrieved chunks contain the relevant information, but they also contain surrounding text that is not relevant to the specific query. Passing the full chunk into the context window wastes space and introduces noise.
Contextual compression extracts only the sentences or passages from each chunk that are directly relevant to the query. The model receives a tighter, higher-signal context and produces correspondingly better answers.
The compression step adds latency. In most production deployments, the improvement in output quality justifies it. For latency-sensitive applications, it can be applied selectively to queries above a certain complexity threshold.
Evaluation and Monitoring
RAGAS and similar evaluation frameworks measure RAG performance at the component level, not just whether the final answer was correct, but whether the retrieval stage performed well independently of generation.
The two metrics that matter most in production are context recall (did the retrieval stage surface all the relevant information that existed in the knowledge base?) and context precision (was all the retrieved information actually relevant?). Low recall means the model is generating answers without the information it needs. Low precision means the context window is being diluted with noise.
Both degrade over time as documents are updated, new content is added, and user query patterns evolve. Continuous monitoring catches this drift before it becomes visible to users.
Attribution and Transparency
A production chatbot that cannot show its sources is a prototype masquerading as a product.
Every response should carry citations: document title, source link or reference, and where applicable, page number or section heading. This serves two purposes beyond compliance. First, it allows users to verify answers independently, which builds trust over time. Second, it makes errors audible: a user who sees the wrong source cited can report it, which is far more useful feedback than a user who simply stops trusting the system.
The Gap Between Prototype and Production
The prototype RAG stack, basic chunking, cosine similarity retrieval, no reranking, no evaluation, works well enough to demonstrate the concept. It fails in production because real document sets are messier, real queries are more varied, and real users have less tolerance for incorrect answers than a demo audience does.
The production stack described above adds meaningful complexity at every stage. That complexity is not incidental. Each component addresses a specific failure mode that appears reliably at scale.
The teams that get production RAG right treat it as a data quality and systems engineering problem, not just a model selection problem. The model is the last variable to optimise. The pipeline it operates within determines whether optimising the model produces any measurable improvement at all.
If you are building a RAG system for internal use and finding that prototype-level accuracy is not holding up under real conditions, the answer is almost always in the indexing or retrieval stage, not in switching to a different model.
If your RAG prototype works in demos but falls short in production, see how we deploy RAG pipelines with production-grade chunking, hybrid retrieval, and reranking built in from day one.
