Building Production RAG Systems with LangChain

Most RAG tutorials stop at "it works on my three test questions." Production RAG has to work on the questions users actually ask, at the volume they ask them, without silently making things up. Here's the architecture I use when building RAG systems with LangChain for real products.

1. Chunking Strategy Determines Retrieval Quality

Most RAG quality problems trace back to bad chunking, not bad models. Guidelines that hold up in practice:

• Chunk by semantic boundary (headings, sections) before falling back to fixed token windows
• Keep chunks small enough for precise retrieval, large enough to preserve context (roughly 300–800 tokens)
• Overlap chunks slightly (10–15%) so answers spanning a boundary aren't lost
• Store source metadata (document, section, URL) on every chunk for citations

2. Embeddings and Vector Storage

Pick embeddings and storage based on scale and ops overhead, not hype:

• pgvector on PostgreSQL — best default if you already run Postgres; no new infra to operate
• Pinecone / Qdrant — worth it at large scale or when you need managed hybrid search
• Use hybrid search (keyword + vector) for queries with exact terms — product names, IDs, error codes

3. LangChain: Use It Where It Earns Its Keep

LangChain is genuinely useful for retriever composition, document loaders, and chaining multiple LLM calls — but a thin custom wrapper is often better than a deep chain of abstractions for a single retrieval-then-generate flow.

const retriever = vectorStore.asRetriever({ k: 6, searchType: "mmr" });
const docs = await retriever.invoke(userQuery);
const context = docs.map(d => `[${d.metadata.source}] ${d.pageContent}`).join("\n\n");
const answer = await llm.invoke(buildGroundedPrompt(context, userQuery));

Reach for LangChain's higher-level chains when you have multi-step agentic workflows. For straightforward retrieve-and-answer, a direct implementation is easier to debug and cheaper to run.

4. Grounding and Hallucination Control

• Instruct the model to answer only from provided context, and to say so when context is insufficient
• Require inline citations back to source chunks — verifiable answers, not just plausible ones
• Re-rank retrieved chunks before generation to push the most relevant context to the top

5. Evaluation: The Step Most Teams Skip

Without evaluation, "we improved the prompt" is a guess. Build a small golden dataset of real questions with expected answers or expected source documents, and score retrieval precision/recall and answer faithfulness on every change — even a spreadsheet-based version beats no evaluation at all.

6. Cost, Latency, and Observability

• Cache embeddings for unchanged documents; re-embed only on content updates
• Log every retrieval + generation call with token counts, latency, and retrieved sources
• Set hard timeouts and graceful fallbacks for model API failures

Bringing It Together

Production RAG is a retrieval system with a language model attached, not a language model with retrieval bolted on. Get the chunking, retrieval, and evaluation right first — the prompt is the easy part.

I'm Sandaruwan Jayasundara — Senior Software Engineer | AI Engineer, building RAG and LLM-backed products for startups and teams worldwide. Explore AI engineering services or get in touch.