A team spends three months and a pile of GPU budget fine-tuning a model so it will answer questions about their product. It ships. A week later the product changes, and the model is confidently wrong again. They had a knowledge problem, and they reached for the one tool that does not fix knowledge problems.
This is the most common and most expensive mistake in applied AI, and it starts with the wrong question. “Should we prompt, use RAG, or fine-tune?” sounds like a question about which tool is best. It is not. The three do not compete; they fix different things. The real question is a diagnosis: what, exactly, is broken?
Three failures, three fixes
Strip away the jargon and there are only three ways a large language model (an LLM, the kind of AI that writes text on command) fails you on a real task.
- It cannot see the facts it needs: your data, this week’s prices, the private document, anything after its training cut-off. The model is not dumb, it is uninformed.
- It will not behave the way you need, every time. The format drifts, the tone wanders, it ignores the rule on the tenth request. It knows the answer; it will not deliver it in the shape you need.
- You asked wrong. It can do the task, but your instructions were vague, your examples thin, your request underspecified.
Each has its own fix, and they are not interchangeable:
- Cannot see the facts, so use RAG (retrieval-augmented generation): before the model answers, search your own documents, grab the passages that match, and paste them into the prompt (the text you send the model). You are handing it an open book.
- Will not behave, so fine-tune: retrain the model’s own internal settings (its weights, the billions of numbers it learned) on a few hundred examples of the behavior you want, until the format and tone are baked in.
- Asked wrong, so fix the prompt: better wording, a few worked examples, a clear description of the output you expect. Hours of work, roughly free.
The trap that costs the most
Take one concrete case: a support assistant for a software company.
It quotes last quarter’s pricing. That is a cannot-see problem, so you give it the current price sheet with RAG. It gets the price right but answers in three rambling paragraphs when you need one strict format your app can read. No amount of retrieval fixes that; it is a will-not-behave problem, so you fine-tune the format in. And if it has the price and the format but the answer is still unhelpful, that is an asked-wrong problem: fix the prompt first, before you spend a rupee on anything else.
Now the trap. Fine-tuning does not teach the model new facts. It shapes behavior. Pour a knowledge base into a model by fine-tuning and it will cheerfully make things up, because you tuned how it talks, not what it knows. The benchmarks are blunt: in one 2024 study of rare, obscure factual questions (the long tail), giving the model the right passages took it from about 17% to 76% correct, while fine-tuning on those same facts helped far less. Facts go in through RAG. Behavior goes in through fine-tuning. Cross that wire and you pay for it twice: once to build the wrong thing, and again to build the right one.
It is not a ladder
The other myth is that these are rungs: start with prompting, graduate to RAG, and once you are serious, fine-tune. That model is wrong, and expensively so.
They are not stages of maturity; they are tools for different jobs. Fine-tuning to fix a knowledge gap is forcing the model to memorize a database that is stale before training even finishes. Using RAG to fix a formatting quirk bolts a whole search system, and the delay it adds (latency), onto something a small behavioral nudge would have solved. Good teams do not climb the ladder. They diagnose, and reach for the rung that matches the break, which is usually the cheapest one.
The bill nobody prices
Fine-tuning has become technically easy, which is exactly why teams underestimate it. Cheap adapters (LoRA, a trick that trains a small add-on instead of the whole model) let you fine-tune an open model (an off-the-shelf base model you download and run yourself) on a single gaming GPU in an afternoon, where full retraining once needed tens of thousands of dollars of datacenter hardware. On a narrow task, that adapter can match full retraining almost exactly.
But the GPU time was never the cost. The cost is everything around it.
- The data. A good fine-tune needs hundreds to thousands of clean, consistent, hand-checked examples. Building that set is weeks of domain-expert time, and it is usually most of the whole bill.
- The eval. Fine-tuning without an automated test (an eval) to catch regressions, the things that quietly used to work breaking, is not production engineering, it is gambling. That harness is real work you build and re-run.
- The retraining treadmill. This is the one nobody draws. Base models are replaced every few months. When the new one lands, your RAG pipeline needs an afternoon of prompt-tweaking, but your fine-tune needs the whole cycle again: revalidate the data, retrain, re-evaluate, redeploy. You did not buy a model. You adopted a maintenance obligation.
There is one more catch that decides it for many teams: you often cannot fine-tune the strongest models at all. As of 2026 the leading hosted models do not let you retrain them, and one major provider is winding its fine-tuning service down for new users. So “we will fine-tune” usually means “we will adopt an open model and run the serving stack (the machinery that keeps a model answering) ourselves,” which is its own decision with its own bill.
When fine-tuning does earn its keep
None of this means never fine-tune. It means fine-tune on purpose. The real case for it looks like this. One team had already pushed prompting and RAG to their limit on a genuinely hard task, turning plain-English questions into database queries, and stalled at around 85% accuracy with 20-second responses. Fine-tuning an open model on their own question-and-query pairs took it to roughly 95% and under 7 seconds. That is fine-tuning doing its actual job: locking in a skill and a response shape that prompting could not hold, on top of retrieval, not instead of it.
The threshold is a checklist, not a feeling. Fine-tune when: prompting and RAG have hit a measured ceiling; the failure that remains is behavioral, not factual; you have the hundreds of clean examples; you have an eval and a rollback plan; and the volume or latency justifies owning a model. Short of that, you are buying an obligation you do not need.
The honest order
So, in order:
- Fix the prompt. Most “the model is bad” problems are underspecified requests. This is hours of work and it is free, so do it first.
- Add RAG when it cannot see your facts. Private, fresh, changing, or citable knowledge lives in retrieval, not in the model. (Whether that retrieval actually works is its own discipline.) And if the facts are scattered across systems in the first place, no model outruns that.
- Fine-tune last, and only the behavior. When prompting and retrieval have topped out and what is left is how the model acts, not what it knows.
- Combine when your evals prove you need both. Fine-tune the behavior, retrieve the facts, not because hybrid sounds mature, but because the tests say each layer earns its place.
Prompt for unclear instructions. Retrieve for missing facts. Fine-tune for fixed behavior. The trick was never picking the fanciest tool. It was diagnosing the problem before you paid to solve it.