RAG vs Fine-Tuning: Which Should You Use for Your LLM App?

Short answer: use retrieval-augmented generation (RAG) when the model needs to know things, especially private, changing or citable information. Use fine-tuning when the model needs to behave differently: a fixed output format, a domain style, or a narrow task done consistently and cheaply at scale. Most LLM applications should start with good prompting, add RAG for knowledge, and consider fine-tuning only when evaluation shows a behaviour gap prompting can't close.

What each technique actually does

RAG

At query time, you search your own data (documents, tickets, product catalogue, database rows), put the most relevant passages into the prompt, and ask the model to answer from them. The model's weights don't change. Updating knowledge means re-indexing a document, which takes seconds. Answers can cite their sources, and you can filter retrieval by user permissions. The engineering details (chunking, embeddings, vector search, reranking) are covered in building production RAG systems with LangChain.

Fine-tuning

You continue training a model on examples of the input and output you want. The model's weights change, so the behaviour is baked in and doesn't need to be described in every prompt. Today this usually means parameter-efficient methods (such as LoRA) on open-weight models, or a provider's hosted fine-tuning API. You need a curated dataset, typically hundreds to thousands of high-quality examples, and a way to re-train and re-evaluate when requirements change.

Decision table

Costs and maintenance

Neither option is free, and the costs land in different places. The comparisons below are qualitative rules of thumb; the actual numbers depend on your provider, model size and traffic.

RAG costs

• Infrastructure: an embedding pipeline, a vector store (pgvector, FAISS, a managed service) and possibly a reranker.
• Per-request tokens: retrieved context makes every prompt longer, which raises cost and latency.
• Ongoing work: keeping ingestion in sync with source systems and tuning chunking and retrieval quality.

Fine-tuning costs

• Data: collecting, cleaning and reviewing examples is usually the largest cost, and it's human time.
• Training and hosting: training runs, plus hosting the model yourself or paying a provider's premium for fine-tuned inference.
• Lifecycle: every time the base model is upgraded or requirements change, you re-train and re-evaluate.

A common pattern: RAG is cheaper to start and to change; fine-tuning can become cheaper per request at high volume if it lets a small model replace a large one on a narrow task.

Three scenarios

Internal knowledge assistant

Employees ask questions about policies, runbooks and product documentation. The content changes every week, different teams may see different documents, and answers need links back to the source so people can verify them. This is a textbook RAG use case. Fine-tuning would add cost and make updates slow without solving any problem the team actually has.

High-volume support ticket triage

Every incoming ticket must be labelled with a category, priority and product area, in a fixed JSON shape, at thousands of tickets per day. There's little external knowledge involved and plenty of historical labelled tickets. A prompted large model works, but a fine-tuned small model trained on past labels can be faster and cheaper per ticket with comparable accuracy, which is exactly what you'd verify on a held-out set.

Regulated report drafting

Drafts must follow a strict section structure and house style, and every figure must come from the client's current documents. Retrieval supplies the facts and citations; if prompting can't hold the structure and style consistently, a fine-tuned generator handles the form. This is where a hybrid is justified, and where an evaluation set that checks both structure and groundedness earns its keep.

Latency, privacy and vendor lock-in

A few non-functional factors often decide the question:

• Latency: RAG adds a retrieval step and longer prompts. Usually tens to a few hundred milliseconds for retrieval, but it adds up in multi-step agents. A fine-tuned small model can be noticeably faster.
• Privacy: with RAG, sensitive data stays in your own store and is deleted by deleting the document. Data used for fine-tuning is baked into weights, which makes "right to be forgotten" requests much harder.
• Portability: a RAG pipeline can switch to a new base model in a day. A fine-tuned model ties you to that base model until you re-train.

Common mistakes

• Fine-tuning to add knowledge. The model may memorise some facts, but you can't cite them, update them cheaply or stop it blending them with what it already believed.
• Blaming the model for a retrieval problem. If the right passage isn't retrieved, no amount of fine-tuning the generator will fix it. Measure retrieval separately.
• Skipping the prompt baseline. Modern models with structured output and a few examples often meet the requirement without either technique.
• No evaluation set. Without one, you can't tell whether a change helped.

The hybrid approach

RAG and fine-tuning aren't mutually exclusive. Useful combinations:

• Fine-tuned generator, RAG for facts. The model learns your answer structure and tone, while retrieval supplies current, citable content.
• Fine-tuned embeddings or reranker. Training the retrieval side on your domain's query/document pairs can improve recall where domain vocabulary is unusual.
• Fine-tuned small model for routing. A cheap classifier decides which retrieval index, tool or prompt to use before the main model runs.

In a RAG-based chat assistant like the Aiden project listed on my projects page, built with LangChain, OpenAI and FAISS, document Q&A and conversation memory come from retrieval and application state rather than from changing model weights. That's the default I'd recommend for most knowledge-heavy products.

A practical decision process

1. Write 50–100 realistic test questions with expected answers or acceptance criteria.
2. Build a prompt-only baseline and score it.
3. If failures are about missing or outdated knowledge, add RAG and score again.
4. If remaining failures are about format, tone or consistency, try structured output and better examples.
5. Only if a measurable behaviour gap remains, or per-request cost at scale is a problem, fine-tune and compare on the same set.

The measurement side is covered in detail in how to evaluate LLM and RAG applications in production.

Frequently asked questions

Is RAG better than fine-tuning?

Neither is better in general. RAG is better for giving a model access to specific, changing or private knowledge with citations. Fine-tuning is better for changing behaviour: output format, tone, style or a narrow task done consistently. Many products need RAG first and never need fine-tuning.

Can fine-tuning teach a model new facts?

It can to some degree, but it is an unreliable way to do it. Facts learned through fine-tuning are hard to update, can't be cited, and can be blended or hallucinated. If the requirement is 'answer from our documents', retrieval is the more dependable and maintainable choice.

Which is cheaper, RAG or fine-tuning?

RAG has low upfront cost but adds retrieval infrastructure and longer prompts on every request. Fine-tuning has upfront data-preparation and training cost, but can lower per-request cost if it lets you use a smaller model or shorter prompts. At high volume on a narrow task, fine-tuning can pay back; for most early products, RAG is cheaper overall.

When should I combine RAG and fine-tuning?

Combine them when you need both fresh, citable knowledge and a consistent behaviour that prompting can't reliably achieve, for example a fine-tuned model that follows a strict answer format or domain style while still answering from retrieved documents. Start with RAG and add fine-tuning only when evaluation shows prompting has hit its ceiling.

What should I try before either?

Prompt engineering with clear instructions, structured output and a handful of good examples. It's the cheapest option, it's quick to iterate, and it sets the baseline that RAG or fine-tuning must beat on your evaluation set.

Need help choosing?

If you're designing an LLM feature and want a second opinion on the architecture, see my AI engineering services or get in touch.