Basic RAG is straightforward: embed a query, find similar documents, stuff them into a prompt, and generate. It works for demos. It fails in production. Here’s what separates toy RAG from production RAG.
Chunking Strategies
The way you split documents matters more than the embedding model you choose. Fixed-size chunks with overlap is the simplest approach, but semantic chunking — splitting at natural boundaries like paragraphs or sections — preserves meaning far better.
The chunk size tradeoff is brutal: small chunks give you precision but lose context, large chunks preserve context but introduce noise. The sweet spot for most domains is 256-512 tokens with 50-100 token overlap, but you must benchmark on your own data.
A Production RAG Pipeline
interface RAGConfig {
chunkSize: number;
chunkOverlap: number;
topK: number;
rerankerModel: string;
similarityThreshold: number;
}
async function ragPipeline(query: string, config: RAGConfig): Promise<string> {
const queryEmbedding = await embed(query);
const candidates = await vectorSearch(queryEmbedding, config.topK * 3);
const reranked = await rerank(query, candidates, config.rerankerModel);
const filtered = reranked.filter(
(doc) => doc.score >= config.similarityThreshold
);
const context = filtered.slice(0, config.topK).map((d) => d.text).join("\n\n");
return generate(query, context);
}Reranking Changes Everything
Raw cosine similarity is a poor relevance signal. Cross-encoder rerankers like Cohere Rerank or ColBERT dramatically improve retrieval quality. In our benchmarks, adding reranking improved answer accuracy by 23%.
Evaluation
You can’t improve what you don’t measure. Track retrieval precision, answer faithfulness, and answer relevance across a held-out test set. Build this evaluation loop early — it’s the foundation every improvement depends on.