RAG Clearly Explained
The mental model every engineer should have.
Photorealistic Faces for AI Agents. 44% More Usage.
Presented by Anam
Anam gives AI agents a photorealistic face, increasing usage by 44% on average. For many use cases, a face is a far more natural way to interact with AI than chat or voice , think sales agents, recruitment screens, and interactive learning. $5,000 for the best demo built with Cara-4, July 14–26.
RAG Clearly Explained
Your model can explain concepts from years ago but struggle with last month’s product launch or your company’s internal docs. The problem isn't the model. It's that its knowledge is frozen in its training data.
As teams started connecting LLMs to internal knowledge bases, support documentation, and constantly changing business data, keeping answers up to date became a new challenge.
Retrieval-Augmented Generation (RAG) solves this by retrieving relevant information at query time and providing it to the model as context. That keeps answers current without retraining.
What RAG actually is
RAG is a system pattern for giving an LLM the right context before it answers.
Instead of asking the model to rely only on what it learned during training, the application first retrieves relevant information from external sources. Then it passes that information to the model so the answer is grounded in real information.
That distinction matters because RAG is often confused with related ideas:
RAG vs fine-tuning → Fine-tuning changes the model’s weights. RAG keeps the model the same and brings in fresh information at runtime.
RAG vs plain search → Search returns documents. RAG uses those documents to generate a clear answer.
RAG vs vector search → Vector search is one retrieval method. RAG can also use keyword search, SQL queries, graph traversal, or other data access patterns.
The retrieval step can be simple or advanced. The model might retrieve once before answering, or retrieve again as the answer develops.
The core idea is simple: find the right information first, then let the model answer with that information in context.
How RAG works
A RAG system has two flows: one for preparing knowledge and one for answering questions.
The preparation flow happens before any user query. Documents are extracted, cleaned, split into chunks, enriched with metadata, and stored in a searchable index.
A chunk is a small piece of content, such as a paragraph, section, or code block. Instead of searching an entire document, the system searches these smaller pieces, making it easier to find the exact information a user is looking for.
When a user asks a question, the query flow begins. The system retrieves relevant chunks, filters them, optionally reranks them, and passes the best information to the model as context.
Here is a typical request flow using Basic RAG and Vector Retrieval:
RAG is not a single model call. It is an information pipeline that finds the right information before the model starts generating.
Main types of RAG
Not all RAG systems work the same way. The biggest differences are how information is retrieved and how much reasoning happens before an answer is generated.
For most teams, Basic RAG is the best place to start. It is easier to build, observe, and debug.
Agentic RAG becomes useful when a question requires multiple lookups or reasoning steps. For example: “Compare this customer’s contract with our refund policy and draft a response.”
That request may require several document lookups, comparisons, and reasoning steps before an answer can be generated.
Retrieval methods
Many teams hear “RAG” and immediately think “vector databases.” In reality, vector search is just one way to retrieve information.
Hybrid retrieval is often the safest default because real questions usually need both. A user might include an exact invoice ID while asking a broader policy question. Keyword search catches the ID. Vector search catches the meaning.
These choices are independent. For example, a system might use Basic RAG with Hybrid Retrieval, or Agentic RAG with Graph Retrieval. One defines how the system operates. The other defines how it finds information.
What usually breaks
A RAG system is only as good as the information it retrieves. Even a powerful model struggles when the context is incomplete, irrelevant, or outdated.
The most common problems are:
Poor chunking → The information is split across chunks, so only part of the information gets retrieved.
Weak metadata → The system cannot filter by tenant, date, permissions, or other important attributes.
Poor retrieval quality → Relevant information exists, but the retriever fails to find it.
Too much retrieval → Irrelevant chunks fill the prompt and bury the useful information.
Stale indexes → Outdated or deleted content continues to appear in answers.
Unsafe retrieval → Untrusted content injects instructions that interfere with the model.
That last one is especially important.
Retrieved documents are external input and shouldn’t always be trusted. The model should follow system instructions, not instructions hidden inside a webpage, PDF, email, or document.
When not to use RAG
RAG works best when you need current information, private data, source citations, or knowledge that changes frequently.
But it is not always the right tool.
Behavior problems → If you need a specific writing style, output format, or classification behavior, fine-tuning is often a better fit because the challenge is behavior, not missing information.
Simple lookups → If a user enters an order ID and needs a delivery date, a direct database query is simpler, faster, and more reliable.
Small knowledge sources → If the entire document fits comfortably in the model’s context window, retrieval may add unnecessary complexity. Passing the document directly to the model is often enough.
Not every AI problem is a retrieval problem. RAG shines when information is missing, changing, or hard to find.
Wrapping up
RAG makes a model more useful without changing its weights. Instead, it changes what the model considers before it answers.
That sounds simple, but it is hard to ship well. Chunking, retrieval quality, permissions, and freshness all need to work before the model generates anything.
The difference between a useful answer and a hallucination often comes down to one thing: whether the right information was retrieved before the model started generating.
👋 If you found this useful → Like + Restack to help others learn system design.