Softechinfra

RAG vs fine-tuning: which does your AI project need?

Retrieval-augmented generation (RAG) and fine-tuning are both ways to make a language model better at your task, but they fix different problems. RAG adds the right information to the prompt at the moment the model answers. Fine-tuning trains the model on your examples so it behaves differently. This page draws on the model vendors’ own guidance (OpenAI, Google and Anthropic) and the original RAG paper, checked on 24 Oct 2026.

By Softechinfra Editorial · Published 24 Oct 2026 · Facts checked on 24 Oct 2026

The short answer

  • Use RAG

    when the model needs facts it doesn’t have: your documents, policies, product data, anything that changes or that answers must cite. OpenAI’s accuracy guide calls this “context optimization”, which maximises response accuracy.

  • Use fine-tuning

    when the model has the knowledge but doesn’t behave the way you need: a fixed output format, a tone, consistent labels, or a smaller model doing one narrow task. OpenAI calls this “LLM optimization”, which maximises consistency of behaviour.

  • Start with neither

    until you have tried clear prompts against a set of test questions. If your knowledge base is under about 200,000 tokens (about 500 pages), Anthropic notes you can put all of it in the prompt. And check your provider still offers fine-tuning: OpenAI has closed it to new organisations.

Side by side

RAG and fine-tuning compared, from vendor documentation (checked 24 Oct 2026)
FactorRAGFine-tuning
What changesSources 1, 5The prompt. Relevant passages are retrieved and added before the model answers.The model. Training on your labelled examples adapts its weights to your task.
Problems it fixesSources 1The model lacks knowledge that wasn’t in its training set, its knowledge is out of date, or it needs your private information.Inconsistent results or formatting, the wrong tone or style, or reasoning that isn’t followed consistently.
When facts changeSources 1Update the documents and the search index; the model stays the same.Rerun the fine-tuning and manage new training and validation sets.
Showing sourcesSources 9The retrieved passages can be shown with the answer.No built-in way. The RAG paper lists providing provenance for knowledge stored in a model’s parameters as an open research problem.
Data you needSources 1, 5Your documents, kept current, and a set of test questions to measure retrieval.Labelled examples of good outputs. OpenAI suggests starting with 50+; Google’s guide says “think 100 examples or more”.
Running costSources 2A search step on every request, and longer prompts that carry the retrieved text.Training runs up front; then shorter prompts with fewer examples, which saves tokens at scale and can lower latency.
What goes wrongSources 1The wrong context, or too much irrelevant context, which drowns out the real information and causes hallucinations.Training examples that don’t match what the model sees in production, and overfitting, which a hold-out set helps you detect.
Vendor availabilitySources 4, 6, 3, 5Built into major platforms, for example OpenAI’s file search tool and Google’s RAG Engine.OpenAI stopped new fine-tuning for organisations that had never used it on 7 May 2026; existing customers can’t create new jobs from 6 Jan 2027. Google offers supervised fine-tuning for listed Gemini models.

Why one can beat the other

OpenAI’s accuracy guide includes a worked example: correcting Icelandic sentences. GPT-4 scored 62 (BLEU) with no examples and 70 with three examples in the prompt. Fine-tuned on 1,000 examples, it scored 87. Adding RAG on top dropped it to 83, because this was a behaviour problem, and the extra context added noise.

The same guide notes that many of OpenAI’s largest customer deployments used only prompt engineering and RAG. Its main advice is to “squeeze as much accuracy out of basic methods as you can before reaching for more complex RAG or fine-tuning.”

Long prompts aren’t free either. The guide warns that putting everything in a long context “promises a much simpler retrieval process” but comes “at a cost in accuracy”, so test it with your own questions before you rely on it.

Using both

They stack. OpenAI says that if your tests show problems with both knowledge and behaviour, you may end up with fine-tuning plus RAG, and that you should fine-tune with RAG examples in the training data so the model learns how to use retrieved context.

When the other option is better

When fine-tuning is the better choice

  • A narrow, repeated task with a clear right answer, such as classification or extraction, and you have hundreds of good examples.
  • Every output must follow the same format, and prompting alone doesn’t hold it.
  • Cost or speed at scale: a fine-tuned smaller model can replace a larger one, and shorter prompts save tokens.
  • Your provider still offers fine-tuning for the model you want. Check this first.

When RAG is the better choice

  • The facts change: prices, stock, policies, tickets, documents.
  • Answers must show where they came from.
  • You have documents but no labelled examples of ideal answers.
  • Different users may see different documents, so access has to be checked at the moment of answering.

When you need neither

  • Your knowledge base is small enough to put in the prompt; Anthropic gives about 200,000 tokens (about 500 pages) as the point below which that is the simplest option.
  • Clear instructions and a few examples already pass your test questions.

Where we fit

We build knowledge assistants that answer from a company’s own documents and cite their sources, so we lean towards RAG; see AI chatbots and knowledge assistants. That is why this page leans on the vendors’ own guidance, so you can check our reading against it.

For the work that turns a RAG demo into a production system, see our production RAG checklist.

Questions people ask

Is RAG better than fine-tuning?

Neither is better in general; they fix different problems. RAG gives the model information it lacks, such as your documents or recent data. Fine-tuning changes how the model behaves, such as its format, tone or consistency on one task. Test with your own questions to see which problem you have.

Can fine-tuning teach a model new facts?

To a degree: Anthropic’s 2024 announcement of fine-tuning for Claude 3 Haiku described it as encoding company and domain knowledge. But OpenAI’s guidance treats missing, out-of-date or private knowledge as a context problem for RAG, and facts learned in training can’t be updated without retraining or traced to a source.

Is RAG cheaper than fine-tuning?

They cost money in different places. RAG adds a search step and longer prompts to every request. Fine-tuning costs training runs up front, then can save tokens with shorter prompts or a smaller model. Which is cheaper depends on your volume and how often the facts change.

Can I use RAG and fine-tuning together?

Yes. OpenAI notes the two stack: RAG supplies knowledge and fine-tuning fixes behaviour. If you fine-tune a model that will use retrieved context, include retrieved context in the training examples.

Do I need RAG if the model has a long context window?

Not always. Anthropic notes that a knowledge base under about 200,000 tokens, roughly 500 pages, can go straight into the prompt. Beyond that, or when the content changes often or answers must cite sources, retrieval is usually the better fit. Long prompts can also cost accuracy, so test either way.

Can I still fine-tune OpenAI models?

Only if your organisation is an active fine-tuning customer. Since 7 May 2026, organisations that had never fine-tuned can’t create jobs; since 2 July 2026, neither can those that haven’t used a fine-tuned model in the past 60 days; and active customers lose the ability on 6 Jan 2027. Fine-tuned models keep working until their base models are deprecated. Google still offers supervised fine-tuning for listed Gemini models.

How we checked

Checked on 24 Oct 2026.

We read each vendor’s documentation on 24 Oct 2026. Fine-tuning availability is changing fast, so check the vendor’s current page before you plan around it.

The worked example and the advice on when to use each technique are OpenAI’s, from its accuracy guide. The 200,000-token figure is Anthropic’s, from its September 2024 article on contextual retrieval. The point that fine-tuning can encode domain knowledge is from Anthropic’s July 2024 Claude 3 Haiku announcement.

How we made this: drafted with AI help from the sources below, then each claim was checked against its source.

Sources

  1. OpenAI: Optimizing LLM accuracy, read 24 Oct 2026
  2. OpenAI: Model optimization, read 24 Oct 2026
  3. OpenAI: Deprecations (self-serve fine-tuning), read 24 Oct 2026
  4. Google Cloud: Introduction to tuning (Gemini), read 24 Oct 2026
  5. Google Cloud: RAG Engine overview, read 24 Oct 2026
  6. Anthropic: Introducing Contextual Retrieval (19 Sep 2024), read 24 Oct 2026
  7. Anthropic: Fine-tune Claude 3 Haiku in Amazon Bedrock (10 Jul 2024), read 24 Oct 2026
  8. Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020), read 24 Oct 2026

Spotted something out of date? Email contact@softechinfra.com. Our editorial policy explains how we check and correct pages.