Mastering Retrieval Augmented Generation (RAG): The Complete End-to-End Guide

From “What is RAG?” to production-grade pipelines, evaluation, and advanced techniques — everything you need in one place. 1. Why RAG? The Problem With “Just an LLM” Large Language Models (LLMs) like GPT-4, Claude, Llama 3, and Gemini are astonishing. They can write essays, generate code, translate languages, and hold coherent conversations. But if you have tried to use them in a real business context, you have probably hit the same three walls that everyone else does: 1.1 The Knowledge Cutoff Wall Every LLM is trained on data up to a certain date. Ask GPT-4 about a product launched last week, and it either says “I don’t know” or — worse — invents an answer. This is fine for chit-chat, catastrophic for enterprise use. 1.2 The Hallucination Wall LLMs are next-token predictors, not truth engines. When they don’t know something, they don’t stop — they guess fluently . A hallucinated legal citation, medical dose, or financial figure isn’t just embarrassing; it’s a liability. 1.3 The Private Data Wall Your internal wiki, customer support tickets, PDFs of contracts, Confluence pages, Jira comments — none of that is in the LLM. And you probably don’t want to fine-tune a model every time a document changes. RAG (Retrieval Augmented Generation) is the elegant, practical, and now-industry-standard answer to all three problems. 2. What is RAG? Retrieval Augmented Generation (RAG) is a technique that combines two components: 1. A Retriever — that fetches relevant information from an external knowledge source (documents, databases, APIs) at query time. 2. A Generator — an LLM that uses the retrieved information as context to produce a grounded, accurate answer. 2.1 A Simple Analogy Imagine a very smart student taking an open-book exam. Without RAG → the student answers from memory alone (may forget, may guess). With RAG → the student first flips to the exact right page of the textbook, reads it, then answers. The student (LLM) is the same. The difference is having the right page open in front of them . 2.2 The One-Line Definition RAG = Retrieve relevant context + Feed it to an LLM + Generate a grounded answer. 2.3 A Minimal Mental Model That’s it. Everything else in this article is a variation, optimization, or evaluation of that flow. 3. Problems That RAG Solves Let’s be concrete about what RAG fixes: RAG doesn’t make the LLM smarter. It makes the LLM better informed — and in enterprise settings, that’s usually what you actually need. 4. Key Benefits of RAG 4.1 Freshness Update a document → the next query uses the new version. No retraining. 4.2 Grounding Answers are anchored to retrieved sources, dramatically reducing hallucinations. 4.3 Traceability Every answer can be shown with its citations. Auditors love this. So do users. 4.4 Cost Efficiency Fine-tuning a 70B model is expensive. Adding a PDF to a vector DB is free. 4.5 Privacy & Control Your data never leaves your infrastructure (if you self-host the vector DB and the model). GDPR / HIPAA-friendly. 4.6 Modularity Swap the retriever, swap the LLM, swap the embedding model — each piece is independent. 4.7 Explainability You can literally show the retrieved chunks that produced the answer. 5. Real-World Business Use Cases RAG isn’t a lab curiosity. It’s already in production at thousands of companies. Here are the patterns you’ll see everywhere: 5.1 Enterprise Search / “Ask Your Docs” An internal chatbot that answers: “ What’s our parental leave policy?” by searching Confluence, Google Drive, and Notion. 5.2 Customer Support Automation Ingest all past tickets, product manuals, and FAQs. Deflect 40–70% of L1 tickets with grounded, cited answers. 5.3 Legal & Compliance Assistants Query thousands of contracts: “ Find all agreements where the termination clause requires more than 90 days notice.” 5.4 Medical & Healthcare Q&A Ground answers in latest medical guidelines and journals — never in the LLM’s training data. 5.5 Financial Research Analyze earnings reports, SEC filings, and news to answer: “ What did the CFO say about margins last quarter?” 5.6 Developer Productivity Code-aware chat over your monorepo: “ How do we implement retries in this codebase?” 5.7 E-commerce Product Search Beyond keyword: “ A waterproof jacket good for hiking in cold rain under $200.” 5.8 Education & Tutoring Personalized tutors grounded in the specific curriculum, not the internet. 6. RAG vs. Semantic Search vs. Fine-Tuning These three are often confused. Let’s untangle them. 6.1 Semantic Search What it is: Find documents whose meaning matches the query (using embeddings) — not just keyword match. Output: A ranked list of documents. Analogy: A very smart Google. 6.2 RAG What it is: Semantic Search + an LLM that reads the results and writes an answer. Output: A natural language answer with (optionally) citations. Analogy: A smart Google that also reads the top links and summarizes them for you . 6.3 Fine-Tuning What it is: Update the LLM’s weights on your domain data so the knowledge is baked in. Output: A new, specialized model. Analogy: Sending your intern to a 6-month bootcamp. 6.4 When to Use What Rule of thumb: Fine-tuning teaches the model new skills. RAG gives the model new knowledge. 7. Architecture of a RAG System A production RAG system has two distinct phases: 7.1 Phase 1 — Indexing (Offline, done once or periodically) 7.2 Phase 2 — Query Time (Online, done per user request) 7.3 The Full Architecture Diagram 8. The RAG Pipeline — Step by Step Let’s walk through each stage with the “why” behind the “what”. 8.1 Step 1 — Load Get the raw text out of PDFs, Word docs, HTML, Markdown, databases, APIs, etc. 8.2 Step 2 — Chunk LLMs have context limits; vector search works best on focused passages. Split the text into ~200–1000 token chunks. 8.3 Step 3 — Embed Convert each chunk into a high-dimensional vector using an embedding model. Similar meanings → nearby vectors. 8.4 Step 4 — Store Save the vectors (plus the original text and metadata) in a vector database that supports fast similarity search. 8.5 Step 5 — Retrieve At query time, embed the user’s question and find the top-k most similar chunks. 8.6 Step 6 — Augment Stuff those chunks into a prompt: “ Answer the question using ONLY this context: {chunks}” . 8.7 Step 7 — Generate Send the prompt to the LLM. Return the answer (plus citations). Every step has knobs. Tuning those knobs is the art of RAG. 9. Chunking and Embedding Deep Dive 9.1 Why Chunking Matters If your document is 100 pages long and you embed it as a single vector, that vector represents an average meaning — useless for precise retrieval. Chunking creates many focused vectors, each representing one idea. 9.2 Chunking Strategies 9.3 Chunk Size Guidelines Too small (< 100 tokens): Loses context. “The company reported a loss.” Which company? Too large (> 1500 tokens): Dilutes relevance. LLM may miss the key sentence. Sweet spot: ~300–800 tokens with 50–100 token overlap. 9.4 Code: Chunking with LangChain from langchain.text_splitter import RecursiveCharacterTextSplitter text = """ Retrieval Augmented Generation (RAG) combines the strengths of retrieval-based and generation-based approaches. It first retrieves relevant documents from a knowledge base and then uses a language model to generate an answer. RAG helps reduce hallucinations because the model is grounded in real sources. It also allows for easy updates: change the documents, and the answers update. """ splitter = RecursiveCharacterTextSplitter( chunk_size=200, chunk_overlap=40, separators=["\n\n", "\n", ". ", " ", ""], ) chunks = splitter.split_text(text) for i, c in enumerate(chunks): print(f"--- Chunk {i} ---\n{c}\n") 9.5 What is an Embedding? An embedding is a list of numbers (e.g., 1536 floats) that represents the meaning of a piece of text. "dog" → [0.12, -0.44, 0.98, …] "puppy" → [0.14, -0.41, 0.95, …] (very close to "dog") "submarine" → [-0.71, 0.03, 0.22, …] (far from "dog") The magic: semantically similar texts → geometrically close vectors. 9.6 Code: Generating Embeddings # pip install openai from openai import OpenAI client = OpenAI(api_key="YOUR_KEY") def embed(text: str) -> list[float]: resp = client.embeddings.create( model="text-embedding-3-small", input=text, ) return resp.data[0].embedding vec = embed("What is Retrieval Augmented Generation?") print(f"Dimension: {len(vec)}") # 1536 print(f"First 5: {vec[:5]}") 9.7 Open-Source Embedding Alternatives # pip install sentence-transformers from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-small-en-v1.5") # free, local, fast vecs = model.encode([ "RAG stands for Retrieval Augmented Generation.", "Dogs are loyal companions.", ]) print(vecs.shape) # (2, 384) Popular open embedding models: `BAAI/bge-large-en-v1.5` - top-tier English `intfloat/e5-large-v2` - strong general-purpose `sentence-transformers/all-MiniLM-L6-v2` - tiny, fast, decent `nomic-ai/nomic-embed-text-v1.5` - long context (8k) 10. Vector Embeddings & Vector Database Indexing 10.1 What is a Vector Database? A vector DB stores vectors and supports Approximate Nearest Neighbor (ANN) search : given a query vector, find the top-k most similar stored vectors — fast, at billion-scale. 10.2 Popular Vector Databases 10.3 Similarity Metrics Cosine similarity — most common; measures angle between vectors. Dot product — fast; good when vectors are normalized. Euclidean (L2) distance — geometric distance. 10.4 Code: Indexing with Chroma # pip install chromadb sentence-transformers import chromadb from chromadb.utils import embedding_functions client = chromadb.PersistentClient(path="./chroma_store") embed_fn = embedding_functions.SentenceTransformerEmbeddingFunction( model_name="all-MiniLM-L6-v2" ) collection = client.get_or_create_collection( name="rag_docs", embedding_function=embed_fn, ) docs = [ "RAG combines retrieval and generation for grounded answers.", "Vector databases store embeddings for similarity search.", "Chunking splits long documents into retrievable passages.", "LLMs like GPT-4 can hallucinate without grounding.", ] collection.add( documents=docs, ids=[f"doc_{i}" for i in range(len(docs))], metadatas=[{"source": "handbook", "idx": i} for i in range(len(docs))], ) results = collection.query( query_texts=["How do we prevent LLM hallucinations?"], n_results=2, ) print(results["documents"]) 11. Understanding the Retrieval Process Retrieval is the most impactful stage of RAG. Garbage retrieval → garbage generation, no matter how good your LLM is. 11.1 The Retrieval Objective Given a query $q$ and a corpus of chunks C = {c_1, c_2, …, c_n}, return the top-k chunks that maximize: relevance(q, c_i) Relevance is usually measured by embedding similarity, but can also include keyword overlap, metadata filters, recency, etc. 11.2 Dense vs. Sparse Retrieval Dense retrieval: Uses embeddings. Great for semantic meaning (“car” ≈ “automobile”). Sparse retrieval: Uses keyword statistics (BM25, TF-IDF). Great for exact matches, product codes, names. Hybrid retrieval: Combines both. Almost always beats either alone. 11.3 Metadata Filtering Vector search doesn’t have to be blind. You can pre-filter by metadata: collection.query( query_texts=["quarterly revenue"], n_results=5, where={"year": 2025, "doc_type": "earnings_call"}, ) This is huge for multi-tenant apps, security scoping, and time-based filtering. 11.4 The Retrieval Trade-Off Typical k = 3–10. 12. How Indexing Works Under the Hood Vector search at scale doesn’t do exact nearest neighbor — that’s O(n) per query and would take seconds for a million vectors. Instead, we use Approximate Nearest Neighbor (ANN) algorithms. 12.1 HNSW (Hierarchical Navigable Small World) The dominant algorithm today. Builds a multi-layer graph: Top layers are sparse (long-range links). Bottom layer contains all points. Search descends from the top, greedily hopping toward the query. Result: logarithmic search time with >95% recall. 12.2 IVF (Inverted File Index) Cluster vectors into buckets. At query time, search only the nearest few buckets. 12.3 Product Quantization (PQ) Compress vectors into small codes to save memory. Often combined with IVF (IVF-PQ) for billion-scale search. 12.4 Practical Takeaway You rarely tune ANN parameters directly, but knowing they exist helps you understand: Why recall isn’t 100%. Why ef_search or nprobe parameters exist. Why raising them improves quality but slows things down. 13. Different Retrieval Methods (with Code) 13.1 Similarity Search (Dense) The default. Embed query → find top-k nearest chunks. results = collection.query(query_texts=["What is RAG?"], n_results=3) 13.2 MMR (Maximal Marginal Relevance) Balances relevance with diversity . Prevents returning 5 nearly identical chunks. from langchain_community.vectorstores import Chroma from langchain_openai import OpenAIEmbeddings vs = Chroma(persist_directory="./cdb", embedding_function=OpenAIEmbeddings()) docs = vs.max_marginal_relevance_search( "How does RAG reduce hallucinations?", k=4, # final results fetch_k=20, # candidates to consider lambda_mult=0.5, # 0 = max diversity, 1 = max relevance ) 13.3 BM25 (Sparse Keyword) # pip install rank_bm25 from rank_bm25 import BM25Okapi corpus = [doc.split() for doc in docs] bm25 = BM25Okapi(corpus) query = "vector database indexing".split() scores = bm25.get_scores(query) top_idx = sorted(range(len(scores)), key=lambda i: -scores[i])[:3] 13.4 Hybrid Search (Dense + Sparse) # Combine BM25 and vector scores with a weight def hybrid_score(bm25_score, vec_score, alpha=0.5): return alpha * vec_score + (1 - alpha) * bm25_score Weaviate, Qdrant, and Elasticsearch support hybrid search natively. 13.5 Multi-Query Retrieval Generate several rephrasings of the user’s query, retrieve for each, then merge. from langchain.retrievers.multi_query import MultiQueryRetriever from langchain_openai import ChatOpenAI llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) retriever = MultiQueryRetriever.from_llm( retriever=vs.as_retriever(), llm=llm, ) docs = retriever.invoke("How does RAG reduce hallucinations?") 13.6 Parent Document Retrieval Embed small chunks for precise matching, but return their larger parent chunks for richer context. 13.7 HyDE (Hypothetical Document Embeddings) 1. Ask the LLM to hallucinate a fake answer to the query. 2. Embed the fake answer. 3. Use that embedding to retrieve. Surprisingly effective — the fake answer often looks more like the target chunk than the raw query. 14. Context Augmentation and Generation 14.1 Prompt Template A minimal RAG prompt: You are a helpful assistant. Answer the user's question using ONLY the provided context. If the answer is not in the context, say "I don't know." Context: {retrieved_chunks} Question: {user_question} Answer: 14.2 Best Practices Number the chunks so the LLM can cite them: [1] … [2] … Explicitly forbid hallucination : “If not in context, say you don’t know.” Ask for citations : “Cite the chunk numbers you used.” Cap total context tokens to avoid overflow. Include metadata (source, date) inside each chunk header. 14.3 Code: Full Augmentation + Generation from openai import OpenAI client = OpenAI() def build_prompt(question: str, chunks: list[str]) -> str: ctx = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(chunks)) return f"""You are a precise assistant. Use ONLY the context below. If the answer is not present, reply "I don't know". Cite sources like [1], [2]. Context: {ctx} Question: {question} Answer:""" def rag_answer(question: str, chunks: list[str]) -> str: prompt = build_prompt(question, chunks) resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], temperature=0.1, ) return resp.choices[0].message.content 15. Building a RAG System — Full Code Walkthrough Now let’s put it all together into a working end-to-end system. 15.1 Setup pip install langchain langchain-openai langchain-community \ chromadb pypdf sentence-transformers tiktoken 15.2 Environment import os os.environ["OPENAI_API_KEY"] = "sk-…" 15.3 A Complete Minimal RAG from langchain_community.document_loaders import PyPDFLoader from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain_openai import OpenAIEmbeddings, ChatOpenAI from langchain_community.vectorstores import Chroma from langchain.chains import RetrievalQA # 1. LOAD loader = PyPDFLoader("company_handbook.pdf") pages = loader.load() # 2. CHUNK splitter = RecursiveCharacterTextSplitter(chunk_size=600, chunk_overlap=100) chunks = splitter.split_documents(pages) # 3. EMBED + STORE embeddings = OpenAIEmbeddings(model="text-embedding-3-small") vectorstore = Chroma.from_documents( chunks, embeddings, persist_directory="./chroma_db", ) # 4. RETRIEVE + GENERATE llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) qa = RetrievalQA.from_chain_type( llm=llm, retriever=vectorstore.as_retriever(search_kwargs={"k": 4}), return_source_documents=True, ) result = qa.invoke({"query": "What is our remote work policy?"}) print("Answer:", result["result"]) print("\nSources:") for doc in result["source_documents"]: print("-", doc.metadata.get("source"), "page", doc.metadata.get("page")) That’s a working RAG system in ~25 lines. Everything else is optimization. 16. LlamaIndex vs LangChain Both are Python frameworks for building LLM apps. Both do RAG well. They differ in philosophy. 16.1 LangChain Strengths: Massive ecosystem, agents, tools, integrations with everything. Weakness: Can feel over-abstracted; API churn. Best for: Complex chains, agent workflows, mixed tool use. 16.2 LlamaIndex Strengths: Laser-focused on RAG. Best-in-class indexing primitives, query engines, evaluation. Weakness: Less flexible for non-RAG tasks. Best for: Data-heavy RAG apps, structured document QA. 16.3 A LlamaIndex RAG Example # pip install llama-index from llama_index.core import VectorStoreIndex, SimpleDirectoryReader from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.core import Settings Settings.llm = OpenAI(model="gpt-4o-mini") Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small") documents = SimpleDirectoryReader("./data").load_data() index = VectorStoreIndex.from_documents(documents) query_engine = index.as_query_engine(similarity_top_k=4) resp = query_engine.query("Summarize the RAG chapter.") print(resp) Notice how much less ceremony there is — that’s LlamaIndex’s design goal. 16.4 Which Should You Pick? Building a RAG-first product? LlamaIndex . Building a multi-tool agent where RAG is one capability? LangChain . Doing both? Use them together — they compose fine. 17. Data Loading in Depth Real-world data is messy. Here’s how to handle it. 17.1 PDFs from langchain_community.document_loaders import PyPDFLoader, UnstructuredPDFLoader # Simple text extraction docs = PyPDFLoader("file.pdf").load() # Better: preserves tables, structure docs = UnstructuredPDFLoader("file.pdf", mode="elements").load() 17.2 Web Pages from langchain_community.document_loaders import WebBaseLoader docs = WebBaseLoader(["https://example.com/blog/post"]).load() 17.3 CSVs / Excel from langchain_community.document_loaders import CSVLoader docs = CSVLoader("data.csv").load() 17.4 Notion, Confluence, Google Drive Each has a dedicated loader in langchain_community.document_loaders. Real production systems usually build incremental sync on top: track last-modified timestamps, re-embed only changed docs. 17.5 Markdown / Code from langchain_community.document_loaders import DirectoryLoader, TextLoader docs = DirectoryLoader("./docs", glob="**/*.md", loader_cls=TextLoader).load() 17.6 SQL Databases Two patterns: 1. Text-to-SQL (agent generates SQL from natural language). 2. Row-to-Document (embed each row’s textual columns). 17.7 Metadata is Gold Always attach metadata during loading: for doc in docs: doc.metadata.update({ "source": doc.metadata.get("source", "unknown"), "team": "engineering", "ingested_at": "2026-07-28", "access_level": "internal", }) Metadata enables filtering, security, and citations. 18. Indexing and Retrieval Implementation Let’s build a slightly more serious index. 18.1 Persistent Chroma Index with Metadata from langchain_openai import OpenAIEmbeddings from langchain_community.vectorstores import Chroma from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader loader = DirectoryLoader( "./corpus", glob="**/*.pdf", loader_cls=PyPDFLoader, show_progress=True, ) raw_docs = loader.load() splitter = RecursiveCharacterTextSplitter( chunk_size=800, chunk_overlap=120, ) chunks = splitter.split_documents(raw_docs) for c in chunks: c.metadata["team"] = "product" c.metadata["version"] = "v2" emb = OpenAIEmbeddings(model="text-embedding-3-large") vs = Chroma.from_documents( chunks, emb, persist_directory="./prod_chroma" ) 18.2 Retrieval with Filters retriever = vs.as_retriever( search_type="mmr", search_kwargs={ "k": 5, "fetch_k": 25, "lambda_mult": 0.6, "filter": {"team": "product"}, }, ) docs = retriever.invoke("What changed in v2?") 18.3 Ensemble Retriever (Hybrid) from langchain.retrievers import EnsembleRetriever from langchain_community.retrievers import BM25Retriever bm25 = BM25Retriever.from_documents(chunks) bm25.k = 5 dense = vs.as_retriever(search_kwargs={"k": 5}) hybrid = EnsembleRetriever( retrievers=[bm25, dense], weights=[0.4, 0.6], ) docs = hybrid.invoke("SKU-2025-A pricing change") Hybrid typically boosts recall on queries containing exact terms (IDs, names, codes). 19. Embeddings and Vector DBs in Practice 19.1 Choosing an Embedding Model Trade-offs to consider: Dimension (384 vs 1536 vs 3072) — bigger = more storage/compute. Context length — how many tokens per chunk it can handle. Domain — general vs. code-specialized vs. multilingual. Cost — API vs. self-host. 19.2 Quick Benchmark Snippet from sentence_transformers import SentenceTransformer, util pairs = [ ("What is RAG?", "Retrieval Augmented Generation combines retrieval and LLMs."), ("What is RAG?", "The Eiffel Tower is in Paris."), ] for name in ["all-MiniLM-L6-v2", "BAAI/bge-small-en-v1.5"]: m = SentenceTransformer(name) for q, d in pairs: e1, e2 = m.encode([q, d]) print(f"{name:35s} sim={util.cos_sim(e1, e2).item():.3f} '{d[:40]}...'") 19.3 Vector DB Selection Cheat Sheet 19.4 Consistency Rule Always use the same embedding model for indexing and querying. Mixing them silently destroys retrieval quality. 20. Augmented Generation — Full Code Let’s write a clean, production-ready augmented generation function. from openai import OpenAI from langchain_community.vectorstores import Chroma from langchain_openai import OpenAIEmbeddings client = OpenAI() vs = Chroma(persist_directory="./prod_chroma", embedding_function=OpenAIEmbeddings(model="text-embedding-3-large")) SYSTEM_PROMPT = """You are a precise, honest assistant. Rules: 1. Answer using ONLY the provided context. 2. If the answer isn't in the context, reply exactly: "I don't know based on the provided documents." 3. Cite sources inline like [1], [2] using the chunk numbers. 4. Be concise. Do not invent facts. """ def format_context(docs) -> str: parts = [] for i, d in enumerate(docs, start=1): src = d.metadata.get("source", "unknown") page = d.metadata.get("page", "?") parts.append(f"[{i}] (source: {src}, page: {page})\n{d.page_content}") return "\n\n".join(parts) def rag(question: str, k: int = 5) -> dict: docs = vs.similarity_search(question, k=k) context = format_context(docs) resp = client.chat.completions.create( model="gpt-4o-mini", temperature=0.1, messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}, ], ) return { "answer": resp.choices[0].message.content, "sources": [ {"source": d.metadata.get("source"), "page": d.metadata.get("page")} for d in docs ], } out = rag("What are the key benefits of RAG?") print(out["answer"]) print("\nSources:", out["sources"]) This pattern — retrieve → format → prompt → generate → return with citations — is the beating heart of virtually every RAG app in production. 21. Evaluating RAG Performance You can’t improve what you don’t measure. RAG evaluation has two axes : 1. Retrieval quality — did we fetch the right chunks? 2. Generation quality — did the LLM produce a good answer given those chunks? 21.1 Building an Evaluation Set Create ~50–200 golden examples: eval_set = [ { "question": "What is our maternity leave policy?", "ground_truth": "26 weeks of paid leave for birthing parents.", "relevant_doc_ids": ["hr_policy.pdf#p12", "hr_policy.pdf#p13"], }, # ... ] 21.2 Retrieval Metrics Hit Rate @ k — was any relevant doc in the top-k? MRR (Mean Reciprocal Rank) — how high was the first relevant doc? Recall @ k — fraction of relevant docs retrieved. Precision @ k — fraction of retrieved docs that were relevant. NDCG — quality-weighted ranking. 21.3 Code: Retrieval Metrics def hit_rate(retrieved_ids, relevant_ids): return int(any(r in relevant_ids for r in retrieved_ids)) def mrr(retrieved_ids, relevant_ids): for i, r in enumerate(retrieved_ids, start=1): if r in relevant_ids: return 1 / i return 0.0 def recall_at_k(retrieved_ids, relevant_ids, k): top = set(retrieved_ids[:k]) return len(top & set(relevant_ids)) / max(len(relevant_ids), 1) 22. Generation Evaluation Metrics Generation is trickier — there’s no single “correct” answer. Common approaches: 22.1 Reference-Based Metrics BLEU / ROUGE / METEOR — n-gram overlap. Weak for open-ended answers. BERTScore — semantic similarity via embeddings. Better. 22.2 Reference-Free / LLM-as-Judge Use a strong LLM (GPT-4, Claude) to grade answers on: Faithfulness — is the answer supported by the context? Answer relevance — does it actually answer the question? Context relevance — was the retrieved context useful? Correctness — vs. a golden reference (if you have one). 22.3 Code: A Simple LLM Judge def llm_judge(question, answer, context, model="gpt-4o"): prompt = f"""Grade this answer on 3 axes (1-5): - Faithfulness (only uses the context) - Relevance (answers the question) - Clarity Question: {question} Context: {context} Answer: {answer} Return JSON: {{"faithfulness": int, "relevance": int, "clarity": int, "reason": str}} """ resp = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=0, ) return resp.choices[0].message.content 23. Choosing the Right Retrieval Method There’s no universally best retriever. Match the method to the query pattern. 23.1 Practical Evaluation Loop methods = { "dense": vs.as_retriever(search_kwargs={"k": 5}), "mmr": vs.as_retriever(search_type="mmr", search_kwargs={"k": 5, "fetch_k": 20}), "hybrid": hybrid, # from earlier } for name, r in methods.items(): hits, mrrs = [], [] for row in eval_set: docs = r.invoke(row["question"]) ids = [d.metadata.get("chunk_id") for d in docs] hits.append(hit_rate(ids, row["relevant_doc_ids"])) mrrs.append(mrr(ids, row["relevant_doc_ids"])) print(f"{name:8s} HitRate={sum(hits)/len(hits):.2f} MRR={sum(mrrs)/len(mrrs):.2f}") Iterate. Measure. Pick what wins on your data. 24. RAG Evaluation with the RAGAS Framework RAGAS is an open-source library that automates RAG evaluation using LLM-as-judge with well-defined metrics. 24.1 Core RAGAS Metrics Faithfulness — is the answer grounded in the retrieved context? Answer Relevancy — does the answer address the question? Context Precision — how much of the retrieved context is relevant? Context Recall — did we retrieve everything needed? Answer Correctness — vs. a ground-truth answer. 24.2 Code: RAGAS in Action # pip install ragas datasets from datasets import Dataset from ragas import evaluate from ragas.metrics import ( faithfulness, answer_relevancy, context_precision, context_recall, ) data = { "question": [ "What is RAG?", "How does chunking help retrieval?", ], "answer": [ "RAG combines retrieval with generation to produce grounded answers.", "Chunking splits documents into focused passages for precise retrieval.", ], "contexts": [ ["Retrieval Augmented Generation combines retrieval with an LLM..."], ["Chunking creates smaller passages so the vector search can find precise matches..."], ], "ground_truth": [ "RAG retrieves external context and feeds it to an LLM for grounded generation.", "Chunking breaks documents into small passages so each embedding captures one idea.", ], } ds = Dataset.from_dict(data) scores = evaluate( ds, metrics=[faithfulness, answer_relevancy, context_precision, context_recall], ) print(scores) 24.3 How to Use RAGAS in Practice 1. Build a curated eval set of ~100 Q/A/context/ground-truth rows. 2. Run RAGAS after every change (new chunker, new embedder, new prompt). 3. Track scores over time — treat regressions like test failures. 25. Advanced RAG: Query Re-Writing Users are messy. Their queries are: Too short → pricing? Too vague → what did they say? Multi-part → What’s the refund policy and how does it compare to competitors? Conversational → *”and what about for enterprise? (needs history) Query re-writing fixes this before retrieval. 25.1 Types of Query Re-Writing 25.2 Why It Matters Better queries → better retrieval → better answers. It’s often the single highest-ROI improvement in a RAG system after basic hygiene. 26. Query Re-Writing — Code 26.1 Simple LLM-Based Rewriter from openai import OpenAI client = OpenAI() REWRITE_PROMPT = """Rewrite the user's question to be a clear, standalone, search-friendly query. Fix spelling, expand acronyms, and add relevant terms. Return ONLY the rewritten query. Original: {q} Rewritten:""" def rewrite(query: str) -> str: resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": REWRITE_PROMPT.format(q=query)}], temperature=0, ) return resp.choices[0].message.content.strip() print(rewrite("pricing?")) # → "What are the current pricing plans and subscription tiers?" 26.2 Multi-Query Expansion EXPAND_PROMPT = """Generate 3 different rephrasings of this question that would help retrieve relevant documents from a knowledge base. Return each on its own line. Question: {q}""" def expand(query: str) -> list[str]: resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": EXPAND_PROMPT.format(q=query)}], temperature=0.3, ) return [line.strip("-•* \t") for line in resp.choices[0].message.content.splitlines() if line.strip()] queries = expand("How do I reset my password?") Retrieve for each, merge, deduplicate. 26.3 Sub-Question Decomposition DECOMPOSE_PROMPT = """Break this complex question into simpler sub-questions. Return one per line. Question: {q}""" def decompose(query: str) -> list[str]: resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": DECOMPOSE_PROMPT.format(q=query)}], temperature=0, ) return [l.strip("-•* \t") for l in resp.choices[0].message.content.splitlines() if l.strip()] print(decompose("What's our refund policy and how does it compare to competitors?")) # → ["What is our refund policy?", "What are our competitors' refund policies?", ...] 26.4 HyDE Implementation HYDE_PROMPT = """Write a short, factual paragraph that would perfectly answer this question, as if you had access to authoritative sources. Question: {q} Paragraph:""" def hyde_retrieve(query: str, k: int = 5): resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": HYDE_PROMPT.format(q=query)}], temperature=0.2, ) fake_answer = resp.choices[0].message.content return vs.similarity_search(fake_answer, k=k) 26.5 Contextual Rewriting (Chat History) CONTEXTUAL_PROMPT = """Given the chat history, rewrite the follow-up question to be a standalone question. Chat history: {history} Follow-up: {q} Standalone question:""" def contextualize(history: str, q: str) -> str: resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": CONTEXTUAL_PROMPT.format(history=history, q=q)}], temperature=0, ) return resp.choices[0].message.content.strip() 27. Query Re-Ranking Even with good retrieval, the top-k list is often noisy. Re-ranking fixes this by using a more expensive but more accurate model to re-score the candidates. 27.1 The Two-Stage Retrieval Pattern Stage 1 (Retriever): Fast, bi-encoder embedding search. Recall-focused. Stage 2 (Re-Ranker): Slow, cross-encoder that scores (query, doc) pairs directly. Precision-focused. 27.2 Why It Works A bi-encoder embeds query and doc independently , then compares vectors. A cross-encoder feeds both together into a transformer, which can reason about their interaction — much more accurate, but too slow to run on millions of docs. Two-stage is the best of both. 27.3 Popular Re-Rankers BAAI/bge-reranker-large — open source, strong. Cohere Rerank — hosted API, very good. ColBERT / ColBERTv2 — late-interaction, fast. LLM-as-reranker — use GPT-4 to score. Highest quality, highest cost. 28. Query Re-Ranking — Code 28.1 Cross-Encoder Re-Ranking (Local, Free) # pip install sentence-transformers from sentence_transformers import CrossEncoder reranker = CrossEncoder("BAAI/bge-reranker-base") def rerank(query: str, docs, top_n: int = 5): pairs = [(query, d.page_content) for d in docs] scores = reranker.predict(pairs) ranked = sorted(zip(docs, scores), key=lambda x: -x[1]) return [d for d, s in ranked[:top_n]] # Two-stage retrieval candidates = vs.similarity_search("What is RAG?", k=30) top_docs = rerank("What is RAG?", candidates, top_n=5) 28.2 Cohere Rerank (Hosted API) # pip install cohere import cohere co = cohere.Client("YOUR_COHERE_KEY") def cohere_rerank(query, docs, top_n=5): texts = [d.page_content for d in docs] resp = co.rerank(model="rerank-english-v3.0", query=query, documents=texts, top_n=top_n) return [docs[r.index] for r in resp.results] 28.3 LLM-as-Reranker RERANK_PROMPT = """Given a query and a passage, rate how relevant the passage is to answering the query on a scale from 0 to 10. Return only the number. Query: {q} Passage: {p} Score:""" def llm_score(query, passage): resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": RERANK_PROMPT.format(q=query, p=passage)}], temperature=0, ) try: return float(resp.choices[0].message.content.strip()) except ValueError: return 0.0 def llm_rerank(query, docs, top_n=5): scored = [(d, llm_score(query, d.page_content)) for d in docs] scored.sort(key=lambda x: -x[1]) return [d for d, s in scored[:top_n]] 28.4 Full Advanced RAG Pipeline def advanced_rag(user_query: str, history: str = "") -> dict: # 1. Rewrite / contextualize query = contextualize(history, user_query) if history else rewrite(user_query) # 2. Multi-query expansion expanded = [query] + expand(query) # 3. Retrieve broadly candidates = [] seen = set() for q in expanded: for d in vs.similarity_search(q, k=15): key = d.page_content[:100] if key not in seen: candidates.append(d) seen.add(key) # 4. Re-rank top_docs = rerank(query, candidates, top_n=5) # 5. Generate context = format_context(top_docs) resp = client.chat.completions.create( model="gpt-4o-mini", temperature=0.1, messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {user_query}"}, ], ) return { "answer": resp.choices[0].message.content, "rewritten_query": query, "sources": [d.metadata for d in top_docs], } This one function embodies most of what a serious RAG system does at query time. Closing RAG is not just a technique — it’s the bridge that turns generic LLMs into trustworthy, up-to-date, domain-specific assistants. Master it, and you master the single most valuable pattern in applied AI today. Mastering Retrieval Augmented Generation (RAG): The Complete End-to-End Guide was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.