What is RAG (retrieval-augmented generation)?
RAG means looking up the relevant passages in your own documents and giving them to the model along with the question, so the answer comes from your material rather than the model's memory. Nothing is retrained. The documents stay yours, and when you update them the answers change immediately.
TL;DR verdict
When a vendor says the model was "trained on your data", ask which they mean. Usually it's retrieval, which is cheaper, updates instantly and can cite its source. That is the better option, so there's no reason to dress it up as training.
| RAG (retrieval) | Fine-tuning | |
|---|---|---|
| What changes | What the model is shown | The model weights |
| Updating a fact | Edit the document | Retrain and redeploy |
| Can cite its source | Yes, the retrieved passage | No |
| Handles a document added today | Immediately | Only after the next training run |
| Good at | Facts, policies, product data | Tone, format, a narrow task |
| Cost driver | Tokens per question | Training runs, then hosting |
| Fails by | Retrieving the wrong passage | Confidently repeating stale training data |
Which we'd pick, scenario by scenario
A support bot over your help center
Every answer is grounded in an article you control, with a link back to it. When the refund policy changes on Tuesday, the bot is correct on Tuesday. RAG was made for this case.
A sales assistant over a price list
Retrieval keeps the numbers current and quotable. We still add a hard rule: the assistant may quote a price only from a retrieved row, never from memory, and says it will check when nothing matches.
An internal search across contracts
People ask in plain language and get the clause plus the document it came from. Here the citation matters more than fluent wording, because the person will open the file anyway.
Where fine-tuning still wins
You want a fixed output shape, or a house tone of voice, on a task where the facts barely move. That is a training problem, not a retrieval one. We rarely combine the two, and only after measuring RAG on its own.
Retrieval is unglamorous, and it wins most of the time for a boring reason: businesses change their facts faster than anyone wants to retrain a model. A price list, a delivery policy, a list of who covers which region. Put those in a document store, retrieve the right rows at question time, and the system is correct the moment someone edits the document.
What people underestimate is that RAG quality depends on search more than on the model. If retrieval hands over the wrong three paragraphs, the best model available writes a fluent wrong answer. So most of our build time on these projects goes into chunking, metadata and evaluation: for the fifty questions customers actually ask, does the right passage come back? We measure that before anyone sees the bot. The technique goes back to a 2020 paper by Lewis and colleagues, and its basic shape has barely changed since; the tooling around it has matured. It pairs naturally with AI agents, which use retrieval as one of their tools. See how we deploy it in AI chatbots.
Sources
Frequently asked
Is RAG the same as training a model on my data?
No, and the difference matters commercially. RAG shows the model your documents at question time and leaves the model untouched; training changes the model itself. RAG is cheaper, updates the moment you edit a document and can point to the source it used.
Does my data leave my systems with RAG?
Only the retrieved passages go to the model provider with each question, and you choose the provider and region. The document store itself can sit in your own infrastructure in the EU. For clients with strict rules we run both the store and the model inside their environment.
Why does a RAG chatbot still get things wrong?
Almost always because retrieval returned the wrong passage, not because the model was weak. The fix is better chunking, better metadata and a test set of real questions, and that is where most of the build effort goes.
How many documents can RAG handle?
The number of documents is rarely the limit; the quality of the search over them is. We have run this over a few dozen policy pages and over tens of thousands of records. The second needs a proper vector store and evaluation, not a bigger model.
What does RAG cost to run?
The running cost is the retrieved passages plus the question and answer, billed per token by whichever model you use, plus a small hosting cost for the document store. Long retrieved contexts are the expensive part, so tighter retrieval is also cheaper.
Which process costs you the most?
Thirty minutes, free. You leave with a concrete fix plan and what the process costs you. The plan is yours, even if you build it without us.
Book a free audit