Production RAG patterns are what separate the impressive demo from the system that actually answers customer questions correctly at 3am. Most teams ship a naive vector search plus prompt template, get 60% accuracy, and then wonder why users abandon the feature. The teams shipping reliable retrieval-augmented generation are using a stack of techniques — query rewriting, hybrid search, reranking, citation enforcement — that compound to push accuracy past 90%. Here are the patterns that earn their complexity.
Table of Contents
Query Rewriting Before Retrieval
The biggest accuracy win in production RAG comes before you ever touch the vector database. User queries are messy, contextual, and often reference prior turns in a conversation. Sending them straight to embedding search returns mediocre chunks.
A small LLM call that rewrites the query into a standalone, well-formed question dramatically improves recall. For multi-turn chat, that rewrite step needs to incorporate conversation history. Anthropic’s documentation on prompt engineering for retrieval covers the techniques in detail. Budget 200ms and 500 tokens for this step — it pays for itself.
Hybrid Search Beats Pure Vector Search
Pure semantic search misses exact keyword matches that users actually type — product SKUs, error codes, function names. Pure keyword search misses conceptual matches. Hybrid retrieval, typically using BM25 alongside dense vectors and combining results with reciprocal rank fusion, consistently outperforms either alone.
Postgres with pgvector plus the built-in full-text search gives you both in one database with no extra infrastructure. For larger scale, dedicated vector databases like Weaviate and Qdrant ship hybrid search natively. The pattern matters more than the implementation choice.
Chunking Strategy Is Most of the Battle
Bad chunking ruins everything downstream. Splitting a markdown document at 1000 character boundaries cuts code blocks in half, separates headings from their content, and produces chunks that lose meaning out of context.
Use semantic chunking — split on document structure (headings, paragraph breaks, code fences) and target 300-800 tokens per chunk with 50-100 token overlap. For tabular data and code, treat each unit as its own chunk regardless of size. Pair chunks with their parent document title and section path as metadata.
Reranking Cuts the Final Set
Vector search retrieves the top 50; a reranker model (Cohere Rerank, BAAI bge-reranker, or a small fine-tuned cross-encoder) scores the relevance of each chunk to the actual query and you keep the top 5-10. The latency cost is 100-300ms; the accuracy gain is substantial.
This is where teams trying to be too clever fail. They skip reranking to save the API call and wonder why irrelevant chunks pollute their context. The reranker is doing different work than the retriever — keep both. Combine with AI-powered cybersecurity practices when handling sensitive document indexes.
Citation Enforcement Builds Trust
Hallucinations destroy user trust in RAG systems faster than any other failure mode. The fix is structural: require the model to cite the chunk ID for every claim, then post-process to verify each cited chunk actually exists and contains supporting text.
Display citations as inline links to source documents in the UI. Users learn to trust the system because they can verify. Internally, log citation rates and uncited claim rates as your primary quality metrics — they correlate better with user satisfaction than any benchmark score. Read the Pinecone RAG learning series for deeper architecture patterns.
Wrap Up
Production RAG patterns reward stacking techniques rather than chasing a single magic bullet. Query rewriting, hybrid search, semantic chunking, reranking, and citation enforcement each contribute incremental accuracy gains that compound. Build evals first, measure ruthlessly, and treat RAG like the search engineering problem it is. Combining this with AI in web design and UX strategies makes for genuinely useful AI features.
Frequently Asked Questions
Do I need a dedicated vector database or is pgvector enough?
pgvector handles 10M+ vectors comfortably with proper indexing (HNSW or IVF). Move to a dedicated vector DB when you need multi-tenancy isolation, hybrid search at scale, or sub-50ms p99 latency at high QPS.
How much should I spend on embeddings?
For most apps, OpenAI’s text-embedding-3-small or Voyage’s voyage-3-lite at fractions of a cent per 1K tokens is plenty. Spend on reranking and the generation model instead.
What chunk size works best?
300-800 tokens with 50-100 token overlap is the right starting range for prose. Code and tables should be chunked semantically by unit, not by token count.
Should I fine-tune embeddings on my domain?
Only after you exhaust other improvements. Query rewriting, better chunking, and reranking typically beat fine-tuned embeddings unless your domain vocabulary is very specialized.
How do I evaluate a RAG system?
Build a labeled eval set of 100-500 queries with known correct answers. Measure retrieval recall@k separately from end-to-end answer accuracy. Track both as you iterate.
