“Should we fine-tune or use retrieval?” is the most common question teams ask us, and it’s usually asked backwards — starting from the technique instead of the property they need. Four questions settle it.
First: does the knowledge change faster than you can retrain? Prices, inventory, tickets, documents — anything with a freshness requirement belongs in retrieval. Fine-tuning bakes knowledge in at training time; if it’s stale a week later, you’ve built a very expensive cache with no invalidation.
Second: is the problem knowledge or behavior? Retrieval adds facts; it cannot add skills. If the model has the facts but formats answers wrong, follows the wrong process, or misuses tools, that’s behavior — and behavior is what fine-tuning changes. No amount of context fixes a model that doesn’t know how to act on it.
Third: can you attribute answers? If your domain needs citations — legal, medical, anything audited — retrieval gives you provenance for free. A fine-tuned model’s knowledge has no receipts.
Fourth: what’s your query volume? Retrieval pays per request forever: index maintenance, extra context tokens, added latency. Fine-tuning pays once. At high volume, the fine-tune that costs more up front is dramatically cheaper per query — our distillation work exists precisely because inference cost dominates at scale.
Most production systems land on both: fine-tune for behavior and the stable core of the domain, retrieve for freshness and provenance. The mistake isn’t picking the wrong one — it’s picking either one before answering the four questions.
