Why Fine-Tuning Is No Longer Your First Choice for Custom AI?

Context engineering, RAG, and agent skills now solve most customization problems — so when does fine-tuning still make sense? Fine-tuning is heavier to build; modern AI systems increasingly rely on lighter, modular customization instead (Source: AI-Generated Image) Is fine-tuning large language models still needed today? Let’s take a well-known legal AI company called Harvey. Back in 2023, they fine-tuned their own model — their own custom AI — in partnership with OpenAI. In blind tests, attorneys preferred this fine-tuned model over the Frontier model at the time, which was GPT-4. They preferred it 97% of the time. A win for fine-tuning: they built a custom AI that lawyers actually preferred over the off-the-shelf leading model. So the lesson is, if a general-purpose model isn’t quite right for some specific use case like legal work, then fine-tune it. Right? What Fine-Tuning Actually Is First, let’s define what fine-tuning actually is. We start with a base model, a base LLM. This is what comes off the shelf — either working with a Frontier lab directly or picking an open-source model. The base model is trained on a massive amount of data; effectively, we scrape information from the internet and use it to train the base model, so there’s a lot of general knowledge baked into the model's weights. Fine-tuning takes that base model and customizes it by continuing its training, but now on a much more focused dataset — some specific documents really focused in one particular area. It might be legal contracts, or an organization’s internal support tickets: the stuff that isn’t sitting around on the internet waiting to be scraped by base models. The result is a new model, a fine-tuned model, and that fine-tuned model incorporates the information from the focused dataset plus all of the weights from the base model, now adjusted with that additional data. This resulting model should get better at certain narrow tasks. That is the wonder of fine-tuning. But in practice, how well does it work? When the Benchmark Flipped Let’s go back to that legal AI company Harvey. In 2025, they created their own legal benchmark to measure how effective their models were at performing particular tasks, and they tested their fine-tuned system against the latest crop of Frontier AI models — the custom model against a bunch of general-purpose Frontier AI models available at the time. The result? Seven of the general-purpose models had now surpassed the company’s custom model on the benchmark — models that had never received any custom legal fine-tuning, yet were still better. Bloomberg saw something similar. They famously trained BloombergGPT from scratch, and later evaluations found GPT-4 and ChatGPT outperforming BloombergGPT on many financial benchmarks. So where does that leave fine-tuning today? Is all that custom training worth doing when big frontier general models keep getting smarter on their own? Why General Models Caught Up To answer that, it’s worth considering how general models have, in many cases, caught up to custom-trained ones. There are a few reasons. One: context windows have got really, really big. The original GPT-3 used a token window of 2K — 2,000 tokens. Today, Frontier models routinely handle much more than that, like 1 million tokens plus of input. So if a model can read, say, 500 pages of legal documents directly in its prompt, then why bake those documents into the weights at all? Just pass the stuff that’s contextually relevant when prompting. Two: reasoning models. They do extended thinking at inference time, working through a problem step by step before answering. So reasoning comes from how hard the model thinks at the moment of the question — at inference time — rather than just from how it was trained months earlier. Three: cheaper inference. Models are getting more efficient and smarter. When the frontier model is constantly getting smarter and cheaper, training a custom version becomes a moving target: by the time the fine-tuned model ships, the next frontier release may have leapfrogged it already. Source: AI-Generated Image Customization Without Touching the Weights General models have gotten better — but if we’re not adjusting weights, how do we make a general model behave like a specialist, like a legal scholar, for example? It turns out there’s a whole stack of customization techniques that don’t touch the model weights at all. The first is RAG, retrieval-augmented generation. Instead of training the documents into the model, the application retrieves — that’s the R in RAG — the documents at query time and then feeds them into the prompt. # Minimal RAG: retrieve relevant chunks at query time, then feed them into the prompt from openai import OpenAI client = OpenAI() def answer_with_rag(question: str, vector_store, k: int = 5) -> str: # R — retrieve the most relevant documents for this specific query docs = vector_store.similarity_search(question, k=k) context = "\n\n".join(d.page_content for d in docs) # A + G — augment the prompt with that context, then generate prompt = ( "Answer the question using only the context below.\n\n" f"Context:\n{context}\n\n" f"Question: {question}" ) resp = client.chat.completions.create( model="gpt-5", messages=[{"role": "user", "content": prompt}], ) return resp.choices[0].message.content In practice, this means the model never “learns” your documents — it simply reads the most relevant ones fresh on every query, so your knowledge base can change without ever retraining a thing. There’s also the consideration of context, specifically context engineering. The idea is that a good prompt is a carefully assembled bundle of context: the system prompt, the relevant data, maybe some format guidelines and the like, all packaged together. # Context engineering: assemble the prompt as a deliberate bundle of context def build_context(system_prompt: str, retrieved_data: str, format_rules: str, user_query: str): return [ {"role": "system", "content": system_prompt}, # who the model should be {"role": "system", "content": f"Relevant data:\n{retrieved_data}"}, # grounding {"role": "system", "content": f"Output format:\n{format_rules}"}, # guardrails {"role": "user", "content": user_query}, # the actual ask ] Notice there’s no training here at all — the “specialisation” comes entirely from how deliberately the prompt is assembled, not from the weights. The third thing to consider is agent skills — those MD files you can create. Skills are folders of files that package up procedural knowledge: basically how to do something, and the tools to use to do it. The model loads them on demand when it sees a task that calls for them. So instead of fine-tuning a model to know how to write SQL queries against a very specific schema, a SQL agent skill can tell the model exactly what to do — and any general-purpose model can use that skill. Here’s what a minimal SQL agent skill might look like — a simple Markdown file that hands the model the schema, the rules, and the tool it needs: --- name: sql-reporting-agent description: Write and run SQL against the analytics warehouse schema. --- # SQL Reporting Skill ## Schema - orders(id, customer_id, amount, status, created_at) - customers(id, name, region, signup_date) ## Rules - Always filter out status = 'cancelled' for revenue queries. - Use explicit JOINs, never comma joins. - Return at most 1000 rows unless asked otherwise. ## Tools - run_sql(query: str) -> table # executes read-only SQL and returns rows ## Procedure 1. Restate the question as a metric + dimensions + time range. 2. Draft the SQL using the schema above. 3. Validate column names against the schema before running. 4. Call run_sql, then summarise the result for the user. So essentially, fine-tuning isn’t the only path to customization. There’s a whole stack of options that work without ever touching model weights. Fine-Tuning Isn’t Free Having all these no-weight options is a good thing, because fine-tuning is not free. In addition to the training run itself, there’s a cost in collecting the examples, evaluating results, and avoiding regressions — plus a cost of maintaining the custom models as the frontier models move on. All of this begs the question: does anyone still need to fine-tune at all? Yes — but for a much narrower set of reasons than back in 2023. When Fine-Tuning Still Makes Sense There’s a modern technique called LoRA , low-rank adaptation, that lets a team fine-tune by training a small adapter that sits on top of an existing base model. We’ve got the base model with its weights, and then this adapter that sits on top of it. Most of the original weights stay locked. In fact, a lot of what gets labeled as fine-tuning in production today is some flavor of LoRA or a related parameter-efficient method. # LoRA fine-tuning: train a small adapter, keep the base weights frozen from peft import LoraConfig, get_peft_model from transformers import AutoModelForCausalLM base = AutoModelForCausalLM.from_pretrained("base-llm-7b") lora_config = LoraConfig( r=8, # rank of the adapter (small = few extra params) lora_alpha=16, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, task_type="CAUSAL_LM", ) model = get_peft_model(base, lora_config) model.print_trainable_parameters() # Only the adapter is trainable; the original weights stay locked. Notice that only the small adapter is trainable while the base model stays frozen — which is exactly why LoRA is so much cheaper and faster than full fine-tuning. Fine-tuning does still make sense in certain situations: Reduced latency , when that’s super important. If a model has to respond in real time — like a voice agent answering a phone call — frontier reasoning models, with all their thinking time, are often too slow. If real-time responsiveness is the constraint, small fine-tuned models might still be the way to go. Distillation. Take a huge frontier model, generate high-quality outputs from it, then fine-tune a much smaller model on those outputs. You’re essentially generating reasoning traces from a large-parameter teacher model and using them to fine-tune smaller models. Reinforcement fine-tuning (RFT). Instead of training only on fixed correct answers, RFT uses a prompt dataset plus a grader: the model samples candidate answers, the grader scores those answers, and training updates the model to make high-scoring answers more likely. The catch is that RFT only really works when the output can be programmatically graded — which is to say, when there’s a definitive right answer. # Reinforcement fine-tuning (RFT): sample answers, grade them, reward the good ones def rft_step(model, prompt: str, expected_ans: str, grader) -> None: candidates = model.sample(prompt, n=4) # sample candidate answers # Programmatically grade each candidate scored = [(ans, grader(expected_ans, ans)) for ans in candidates] # Pseudocode: update the model so higher-scoring answers become more likely update_policy(model, prompt, scored) def exact_match_grader(expected: str, answer: str) -> float: # Works only when there is a definitive right answer return 1.0 if answer.strip() == expected.strip() else 0.0 This is why RFT only works when correctness can be measured programmatically — no grader, no reward signal, no training. A Practical Decision Framework So fine-tuning isn’t dead. From a practical decision framework today, I think of the order going something like this: Start with a base model. Implement prompt and context engineering. If knowledge is fresh or proprietary, add in capabilities for RAG. If the missing piece is procedural knowledge, add in agent skills. Only then reach for fine-tuning — if there’s a specific bottleneck that the rest of this stack can’t solve. But what do you think? Does fine-tuning still have its place? Let me know in the comments. If you found this helpful, consider clapping👏 so others can find it too and follow me for more amazing technical AI content! Continue Reading llama.cpp vs vLLM: How to Actually Run Local LLMs (and Which One to Pick) CLI vs MCP: I Ran the Same Task Through Both. One Used 250 Tokens. The Other Used Over 2,000. Microsoft Says Don’t Install OpenClaw on Your Work Laptop! I Read the Architecture to Find Out Why. Why a 3B AI Model Can Beat a 70B One — It’s Not About Model Size Anymore Prompt Caching Explained: How to Slash LLM Costs and Latency Without Sacrificing Quality References: OpenAI — “Customizing models for legal professionals” (Harvey case study, incl. the 97% preference result) https://openai.com/index/harvey/ “BloombergGPT: A Large Language Model for Finance,” https://arxiv.org/abs/2303.17564 Reporting that GPT-4 outperformed BloombergGPT on financial benchmarks (Queen’s University research) https://www.companieshistory.com/bloomberggpt-statistics/ Why Fine-Tuning Is No Longer Your First Choice for Custom AI? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.