Fine-tuning vs RAG

You have a specialised task or a private knowledge base. You can either teach the model directly (fine-tuning) or hand it the relevant context at inference time (retrieval-augmented generation). They solve different problems and the right pick depends on what changes about your data.

Use fine-tuning when

• You need to change style, format, or behaviour consistently (output structured JSON, follow a specific tone, refuse particular request types).
• Your task has a small, stable set of patterns the model must learn (extracting fields from invoices, classifying support tickets).
• Latency matters and you cannot afford the extra retrieval round-trip.

Fine-tuning bakes the behaviour into the weights. Inference is single-shot.

Use RAG when

• Your knowledge base changes frequently (product docs, internal wikis, news).
• You need citations and provenance.
• The relevant content is too large to fit in context all at once.

RAG keeps the model frozen and updates the index. Adding new documents takes seconds.

The hybrid

In practice production systems mix both:
1. Fine-tune for output shape and refusal behaviour.
2. RAG for factual grounding.

The biggest mistake teams make is fine-tuning on facts. Facts go stale; behaviour does not. Use the right tool for each.

Cost model