The short answer
RAG (retrieval-augmented generation) is a pattern where an agent first searches a defined knowledge source for information relevant to a question, then actually uses that information to compose its answer, instead of relying only on what the model learned during training. It gives the agent updatable knowledge and a traceable source, and it underlies any agent that needs precise knowledge about a specific organisation or product.
RAG explained plainly
A language model alone answers from its general "memory" acquired during training — general, possibly outdated, and knowing nothing about your company's policies or your specific product. RAG adds a step before generation: search a defined knowledge source (documents, FAQs, policies) for the passages most relevant to the question, then pass those passages to the model so it builds its answer on them explicitly. The practical difference: a plausible-but-generic answer versus one grounded in an actual text you can check.
The retrieval loop
- The user's question arrives and is analysed for what actually needs a lookup.
- The knowledge base is searched for the passages most semantically relevant.
- The best passages are passed to the model as added context, not as absolute fact.
- The reply is built on those passages, with the ability to point back to their source.
- If no sufficiently relevant passages exist, the agent should say so rather than invent an answer.
Chunking and sources
Before retrieval, documents are split into chunks of a suitable size: small enough to be precise, large enough to carry understandable context. Poor chunking — passages cut mid-idea, or chunks so long they blend several topics — weakens retrieval more than almost any other part of the system, because the agent cannot find what was never split in a way that preserves its meaning. Source selection matters just as much: one current, trustworthy source beats dozens of contradictory, outdated documents.
When to retrieve vs the alternatives
- Retrieve from the knowledge base when the agent needs relatively stable information: a policy, a procedure, product details.
- Call a tool when it needs to take an action or fetch live data that keeps changing: an account balance, order status, sending an email.
- Ask a human when the decision falls outside what either knowledge or tools together can answer, or when the risk is high.
- Practical rule: knowledge is read, tools act, and a human decides when things are ambiguous or risky.
Table: RAG vs the alternatives
| Approach | When to use | How it updates | Main constraint |
|---|---|---|---|
| RAG | Text knowledge that changes over time | Update the source only | Quality depends on chunking and sources |
| Fine-tuning | Changing the model's fixed style or behaviour | Requires retraining | Costly, unsuited to fast-changing facts |
| Tools | Taking an action or fetching live data | Instant, via the connected system | Not suited to general text knowledge |
| Memory | Information specific to a user or session | Written and updated deliberately | Not a general, trustworthy knowledge source |
Citations and traceability
One of RAG's biggest advantages is that an answer can be linked back to its source: which document or passage it was built on. That makes review and verification possible, something that is impossible when a model answers purely from general memory. Any serious agent system should let you trace an answer to its source, not present it as bare fact.
Common failure modes
- Retrieving from an outdated source with no refresh cycle.
- Poor chunking that strips passages of context and weakens search accuracy.
- Generating an answer despite no sufficiently relevant passages, instead of admitting the information is missing.
- Relying on RAG to perform actions, when it is designed to retrieve text knowledge, not execute operations.