I Built a Production RAG System for Indian Law and Refused to Trust It Until I Measured It

How I turned a stack of legal PDFs into a chatbot that cites the exact section of the law, and why the evaluation scores, not the demo, were the real project. Ask a lawyer in India a simple question “what’s the punishment for cheating?” and you’ll get an answer in thirty seconds. Ask that same question as an ordinary citizen, and you’re staring at a 300-page PDF written in a register designed to keep you out. Most people can’t afford the thirty seconds of a lawyer’s time. So they guess, or they trust a stranger, or they do nothing. That gap is the reason I built Lexora: a chat app where you ask a question about Indian criminal law in plain language and get an answer grounded in the actual law , with the exact section cited BNS · Sec 318 · active so you can verify it yourself. This is the story of how it was built, but more honestly, it’s the story of how I learned not to trust a RAG system that looks like it works and what it took to measure whether it actually did. If you take one thing from this article, let it be this: the demo is a liar, and the evaluation harness is the only thing that tells you the truth. What Lexora actually is Before the internals, the product in three sentences: You ask a legal question. Follow-ups work it remembers the conversation. Every answer carries structured citation chips (e.g. BNS · Sec 103 · active), and each one is validated against the retrieved source , so the model physically cannot invent a citation. It’s a real account-based product: signup/login, saved conversations, history. The corpus is India’s criminal law both the new codes that took effect in 2024 (BNS, BNSS, BSA) and the repealed ones they replaced (IPC, CrPC, IEA), kept for old-vs-new context. That “old vs new” wrinkle matters more than it sounds, and it shaped a lot of the retrieval design. Here’s the whole system at a glance: Now let’s open the box. The RAG brain: retrieval is 80% of the battle A legal question has a nasty property: the keywords matter and the meaning matters, and neither alone is enough. Someone typing “Section 420” needs an exact keyword hit. Someone typing “what if I lied to get money” needs semantic understanding to land on the same offence. So Lexora uses hybrid retrieval: ChromaDB for semantic search over BGE-large embeddings catches meaning. BM25 for keyword search catches exact section numbers and legal terms of art. Both are combined with LangChain’s EnsembleRetriever, then a cross-encoder reranker (bge-reranker-base) re-scores the merged candidates and keeps the best few. That reranking step is underrated. Embedding similarity gets you roughly relevant chunks; a cross-encoder actually reads the query and the chunk together and tells you which ones truly answer it. It’s the difference between “these are in the neighbourhood” and “this is the one.” The router: don’t search what you don’t need Remember the old-vs-new problem? If someone asks about a current offence, dredging up the repealed IPC section pollutes the context. So before retrieval, an LLM router reads a “relationship map” of the corpus and picks metadata filters which act, which status (active vs repealed) so the search runs over the right slice. It’s a small, cheap LLM call that makes every downstream step better. This is a pattern I’d reuse anywhere: let a model narrow the search space before you search. Generation you can’t lie with The answering model (GPT) doesn’t just return prose. It returns structured output: an answer, a list of citations, and an answer_found boolean. Then comes the part I'm proudest of the citation-validation loop: Every section the model cites is checked against the sections that were actually retrieved. If it cites something that isn’t in the context, the answer is rejected and regenerated. A legal assistant that hallucinates a section number is worse than useless it’s dangerous so this loop is non-negotiable. It also produced one of my favourite small bugs. Validation kept failing on correct answers. The cause: the model returned "Section 103" while the metadata stored "103". String equality said "different." The fix was a lesson I keep re-learning: def normalize_section(s): # "Section 103", "Sec. 103", "103" → "103" match = re.search(r"\d+", s or "") return match.group() if match else None # compare normalize_section(cited) against normalize_section(stored) Normalize before you compare. Half of “AI bugs” are really string-formatting bugs wearing a trench coat. Memory ≠ dumping history at the model Conversational memory has a trap. When a user asks “what about culpable homicide?” as a follow-up, you cannot send that raw string to the retriever BM25 and embeddings aren’t an LLM, they have no idea what “what about” refers to. So Lexora runs a query-contextualization step first: an LLM rewrites the follow-up into a standalone question (“what is the punishment for culpable homicide under the BNS?”) before retrieval. The full history still goes to the answering model, but the retriever gets a clean, self-contained query. The lesson: a plain chatbot can dump history at the LLM; a RAG chatbot has to clean the query for the retriever separately. Two different consumers, two different needs. The part nobody blogs about: measuring whether it works Here’s where most RAG tutorials end “look, it answered my question!” and where the actual engineering begins. I did not want to feel like Lexora worked. I wanted a number. So I hand-built a golden dataset of 160 question–answer pairs and ran the pipeline through RAGAS, which scores four things: Faithfulness is the answer actually supported by the retrieved context, or is the model making things up? Answer relevancy does the answer address the question that was asked? Context precision of what we retrieved, how much was actually relevant? Context recall of what we needed , how much did we retrieve? The first run was humbling. Faithfulness was okay, but relevancy came back nan, and precision and recall were mediocre. A demo I'd have happily shown off was, by the numbers, mediocre. Diagnosing from the scores, not from vibes Two things were wrong, and RAGAS pointed at both. 1. The nan was infrastructure, not quality. RAGAS's native embeddings class didn't implement embed_query, so relevancy silently failed to compute. Wrapping the model properly (LangchainEmbeddingsWrapper) fixed it. Worth stating plainly: a nan is not a bad score, it's a broken measurement and confusing the two will send you optimizing the wrong thing. 2. The mediocre retrieval was a chunking problem. My first chunker (RecursiveCharacterTextSplitter) was packing five or six unrelated legal sections into a single chunk. That does two terrible things at once: it dilutes the embedding (one vector trying to represent six offences) and it corrupts the metadata (which section is this chunk even about?). No amount of reranking saves you from bad chunks. Fixing it was not one clean move it was a lot of trial and error. I rewrote chunking to be one section per chunk (a lookahead regex splitting on section boundaries, with the definitions section special-cased), restructured how each chunk’s metadata was built, upgraded embeddings from MiniLM to BGE-large, and tightened the citation handling then re-ran the harness, read the scores, and did it again. And again. Each pass moved a different metric. By the end, the scores had moved where it mattered most faithfulness, the one that decides whether a legal answer can be trusted, reached 93%: Chunking quality dominates RAG quality and getting there is iterative, not a single insight. If your retrieval is bad, fix the chunks before you touch anything else, then measure, then fix again. The integrity lesson that almost fooled me Then a subtle, scary one. Some RAGAS runs came back suspiciously good until I read the logs and found OpenAI rate-limit timeouts silently dropping questions from the average. The hard questions were timing out, getting excluded, and inflating the score. My RAG wasn’t getting better; my evaluation was quietly grading only the easy questions. The fix (a RunConfig with sane max_workers and timeout) was trivial. The lesson was not: "the score went up" means nothing until you check what got excluded from the average. An evaluation you don't audit is just a more sophisticated way of lying to yourself. Wrapping the brain in a product A RAG pipeline in a notebook is a science project. Making it a product was its own arc. API (FastAPI). I wrapped the pipeline in a clean /ask endpoint Pydantic request/response models, HTTPException handling so no stack trace ever leaks to a user, structured citations in the response shaped for the frontend , not the raw RAGAS internals. Auth & data (Supabase). Postgres tables for profiles, conversations, messages, with Row-Level Security on. This produced the best bug of the whole project. Conversation inserts kept failing with an RLS violation even though I was using the service-role key that's supposed to bypass RLS. The culprit: the supabase-py client is a shared singleton, and calling auth methods on it leaked the user's JWT into subsequent table calls, so my "admin" writes were silently running as the limited user. The fix was two separate clients: supabase = create_client(URL, SERVICE_ROLE_KEY) # DB writes supabase_auth = create_client(URL, ANON_KEY) # auth only Know your library’s hidden state. A shared client with mutable auth is a landmine, and the error message (“RLS violation”) pointed nowhere near the real cause. Frontend (Next.js). Landing → auth → chat, in a dark/gold brand, fully mobile-responsive (the chat sidebar collapses into a slide-in drawer). One process habit paid off repeatedly: I built the UI against mock data shaped like the real API first , so wiring the backend later was a swap, not a rewrite. The deployment gauntlet Shipping is where the estimates go to die. Host selection was a live-fire exercise. HuggingFace Spaces made Docker a paid feature the week I tried it. Hetzner’s cheap ARM instances were sold out everywhere, and their x86 8GB was €35/mo my mental price list was months stale. I checked live prices and pivoted to a Contabo VPS (~€5/mo, 8GB), running Docker + a Caddy reverse proxy, with the ML models downloading from the HuggingFace hub at runtime into a cached volume and all secrets living in a .env on the server, never in git. Then the classic. On the deployed frontend, chat just… did nothing. The console showed a mixed-content block: the page (HTTPS) was calling /conversations (no trailing slash), FastAPI was issuing a 307 redirect to /conversations/ but building that redirect URL as http://. Why? Uvicorn was behind Caddy, which terminates TLS, so uvicorn genuinely thought it was serving plain HTTP and had no idea the original request was HTTPS. The fix is one flag: uvicorn app:app --proxy-headers --forwarded-allow-ips=* Now uvicorn trusts Caddy’s X-Forwarded-Proto: https header and builds correct URLs. Behind a reverse proxy, you must tell the app the original scheme, or every URL it generates is subtly, invisibly wrong. Hardening for the open internet. Once it was public, the bots arrived instantly a steady drizzle of requests probing /wp-config.php and /.env (all 404, all normal). I locked CORS to the frontend origin, added rate limiting (/ask at 10/min, auth at 5/min which is as much about protecting the OpenAI bill as protecting the server), and put indexes on the hot database columns. What I’d tell my past self Technical RAG quality lives or dies on chunking and retrieval. Measure it (RAGAS), don’t eyeball it. The retriever is not an LLM. Give it clean, standalone queries. Validate every citation against the retrieved context so the model can’t hallucinate a source. Normalize before you compare. "Section 103" and "103" are the same fact in two costumes. Behind a proxy, forward the scheme or debug mixed-content errors at midnight. Know your library’s hidden state (looking at you, shared Supabase client). Process Ship first, harden second and decide what not to build so v1 actually ships. Build UI against mock data shaped like the real API. Verify against reality a real browser, real logs and always check what your evaluation excluded , not just the headline number. Estimates go stale. Check live prices and availability before you commit. Where it stands Lexora is live at lexora.cvijay.dev a stranger can visit a URL, ask a legal question in plain English, and get back an answer grounded in the actual section of the law, with a citation they can check. That was the whole milestone, and it’s real. Next on the roadmap: response streaming (the biggest remaining UX win), migrating embeddings to cut hosting cost, and expanding beyond criminal law into more domains. But the part I’ll carry into the next project isn’t the stack. It’s the discipline: I stopped trusting the demo the day the evaluation harness told me it was lying. If you’re building RAG and you don’t have a number yet that’s the first thing to build, not the last. If you’re working on legal AI, retrieval evaluation, or just want to compare notes on shipping RAG to production, I’d love to hear from you. I Built a Production RAG System for Indian Law and Refused to Trust It Until I Measured It was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.