Production RAG Pipeline: The Architecture, Chunking and Evaluation Guide

A production RAG pipeline is a system that retrieves the right pieces of your own data and gives them to a language model so it can answer accurately and cite its sources. A demo needs a vector database and one prompt. A production system needs clean ingestion, smart chunking, hybrid retrieval, reranking, grounded generation and, most importantly, a way to measure quality.

This guide walks through each stage, what usually goes wrong, and how to know whether your pipeline is actually good.

TL;DR

• Most bad RAG answers come from bad retrieval, not a bad model. Fix retrieval first.
• Chunk by meaning (headings, sections), not by fixed character count alone.
• Use hybrid search (keyword plus vector) and a reranker before sending text to the model.
• Build a small evaluation set of real questions before you tune anything.
• Track cost and latency per request from day one.

What is RAG, and why does it fail in production?

RAG stands for Retrieval-Augmented Generation. Instead of asking a model to answer from memory, you search your own documents, pick the best passages, and ask the model to answer using only those passages.

It fails in production for a simple reason: demos use five clean documents, while real systems use thousands of messy ones. PDFs have broken tables, websites change, FAQs contradict each other, and users ask questions in ways nobody planned for.

The six stages of a production RAG pipeline

1. Ingestion: clean data beats clever prompts

Everything downstream depends on what you put in. Before embedding anything:

• Remove navigation menus, cookie banners, footers and repeated boilerplate.
• Deduplicate near-identical pages.
• Keep the source URL, title, date and document type with every document.
• Plan for updates. If your data changes weekly, ingestion must be a scheduled, repeatable job, not a one-time script.

A common mistake is re-embedding everything on every run. Track a content hash per document and only re-process what changed. This saves money and keeps your index consistent.

2. Chunking: split by meaning

Chunking decides what the model can see. Chunks that are too small lose context. Chunks that are too large bury the answer in noise and waste tokens.

Good practice:

• Split on headings and paragraphs first, then enforce a maximum size.
• Add a small overlap between neighbouring chunks so ideas are not cut off.
• Prepend the document title and section heading to each chunk, so a chunk still makes sense on its own.
• Treat tables, code and FAQs as special cases. One FAQ question and its answer should usually be one chunk.

There is no universal best chunk size. Test two or three sizes against your own questions and keep the one that retrieves best.

3. Indexing and metadata

Store every chunk with metadata such as source, category, language, date and access level. Metadata lets you filter before searching, for example "only current pricing documents" or "only documents this user is allowed to see". This is also how you keep private data private in multi-tenant systems.

4. Retrieval: use hybrid search

Vector search finds meaning, but it can miss exact terms like product codes, error messages or names. Keyword search (BM25) finds exact terms, but it misses paraphrases. Hybrid search combines both and is the safer default for most business content.

Also consider:

• Query rewriting. Turn "what about the cheaper one?" into a standalone question using chat history.
• Top-k tuning. Retrieve a wider set (for example 20 to 50 candidates), then narrow down with a reranker.

5. Reranking: the cheapest quality upgrade

A reranker reads the question and each candidate passage together and scores how well they match. It is slower than vector search but far more precise, so you run it only on the shortlist. Sending the best 3 to 6 passages to the model instead of the first 10 usually improves answers and lowers token cost at the same time.

6. Generation: keep the model grounded

Your prompt should make three things explicit:

1. Answer only from the provided context.
2. If the context does not contain the answer, say so instead of guessing.
3. Include the source of each claim so users can verify it.

Ask for structured output (for example JSON with answer and sources) when another system consumes the result. For high-stakes answers, add a human approval step before anything reaches a customer.

How to evaluate a RAG pipeline

This is the step most teams skip, and it is the one that separates a toy from a product.

Build a golden set. Collect 50 to 100 real questions from users, support tickets or search logs. For each, note the correct answer and the document it should come from.

Then measure two things separately:

• Retrieval quality. Did the correct passage appear in the top results? (Hit rate, recall@k.)
• Answer quality. Is the answer correct, complete and faithful to the retrieved text? You can use human review, or an LLM as a judge, but spot-check the judge yourself.

Run this set every time you change chunking, embeddings, prompts or models. If a change improves one question and breaks five, you will see it immediately instead of hearing about it from users.

Cost and latency

Every stage adds time and money: embedding, search, reranking and generation. Log these per request:

• Tokens sent and received
• Time spent in each stage
• Cache hits

Useful savings include caching frequent answers, sending fewer but better passages, and using a smaller model for simple questions. For a deeper look, read 7 practical habits to control AI costs. If your RAG system is behind a voice interface, latency matters even more, which I cover in how to build a production voice AI agent.

Common RAG mistakes

1. Tuning the prompt before fixing retrieval. If the right passage is not retrieved, no prompt can save the answer.
2. No evaluation set. Without one, every change is a guess.
3. Ignoring stale data. Old pricing or policies in the index produce confident wrong answers.
4. No metadata filtering. Users see content they should not, or irrelevant content crowds out relevant content.
5. Stuffing the context window. More text is not better. Noise hurts accuracy and raises cost.
6. No fallback. When confidence is low, route to a human or say "I do not know".

Production RAG checklist

• Sources cleaned, deduplicated and tagged with metadata
• Scheduled, incremental ingestion
• Chunks keep title and heading context
• Hybrid retrieval with a reranker
• Prompt forces grounded answers with sources
• Golden question set and repeatable evaluation
• Cost and latency logged per stage
• Human or fallback path for low-confidence answers

Frequently asked questions

What is the difference between RAG and fine-tuning?

RAG gives the model fresh knowledge at question time by retrieving your documents. Fine-tuning changes the model's behaviour or style through extra training. For company knowledge that changes often, RAG is usually the better and cheaper choice.

Which vector database should I use?

Choose based on scale, hosting preferences and filtering features. For many projects, a managed service or Postgres with a vector extension is enough. Good chunking and retrieval matter far more than the database brand.

How many chunks should I send to the model?

Start with 3 to 6 reranked chunks and test. More is not automatically better.

How do I reduce hallucinations in RAG?

Improve retrieval, instruct the model to answer only from context, require sources, and add a "not found" fallback. Then measure with an evaluation set.

Final thoughts

A reliable RAG system is built stage by stage: clean data, meaningful chunks, hybrid retrieval, reranking, grounded answers and constant measurement. Get these right and the model almost looks smarter than it is.

Building a RAG system or automation for your business? Get in touch.