Englishবাংলা
SubscribeWrite
Latest
Binary search: patterns, templates and common pitfallsDocker's $250 Cloud Sandboxes credit: what it covers and what it does notTwo pointers and sliding window: patterns with examplesArrays and hash maps: the interview patterns you must knowBig-O notation explained: time and space complexity for interviewsSoftware testing basics: the test pyramid and what to testClean code and code review habits that make you a better developerREST API design best practices: URLs, status codes and errors

RAG explained: answer questions from your own documents

RAG gives a language model the right documents at question time, so answers are grounded in your data instead of guesswork.

••3 min read
A question retrieving documents that feed a model's answer

A language model only knows its training data, which may be old and does not include your private documents. RAG (retrieval-augmented generation) fixes this: at question time, the system retrieves relevant passages from your own content and gives them to the model to answer from.

The two phases

Indexing (done ahead of time)

  1. Collect documents: help articles, PDFs, wiki pages, tickets.
  2. Clean them and split into chunks.
  3. Create an embedding for each chunk and store it with the text and metadata (see the embeddings article).

Answering (every question)

  1. Take the user's question, optionally rewrite it to be clearer.
  2. Retrieve the most relevant chunks (vector, keyword or hybrid search), maybe re-rank them.
  3. Build a prompt containing the instructions, the retrieved chunks and the question.
  4. The model writes an answer based on those sources, ideally with citations.

A simple prompt

Answer the question using only the sources below.
If the sources do not contain the answer, say "I don't know based on the available documents."
Cite the source number for each claim, like [1].

Sources:
[1] (Refund policy) Refunds are available within 30 days of purchase...
[2] (Shipping) Orders ship within 2 business days...

Question: Can I get a refund after 45 days?

Why use RAG

  • Fresh and private knowledge without retraining the model.
  • Fewer hallucinations, because the answer is anchored to text.
  • Citations, so users can verify.
  • Easy updates: change a document, re-index it, and answers change.
  • Access control is possible at retrieval time.

RAG vs fine-tuning

  • RAG adds knowledge at question time. Best for facts, documents and changing information.
  • Fine-tuning changes the model's behaviour or style (format, tone, a specialised task). It is poor at reliably storing facts and is slower to update.
  • They can be combined, but start with RAG and good prompts.

Where RAG fails

Most "bad AI answers" are retrieval problems, not model problems.

  • The right chunk was not retrieved: poor chunking, weak embeddings, wrong filters, or the question uses different words. Try hybrid search, query rewriting and re-ranking.
  • Too much or noisy context: irrelevant chunks distract the model. Retrieve fewer, better chunks.
  • The answer spans several documents: one chunk is not enough. Retrieve more or use multi-step retrieval.
  • Outdated or conflicting documents: keep dates and versions in metadata and remove stale content.
  • The model ignores sources: make the instruction firm and ask for citations and "I don't know".
  • Tables, PDFs and scans extract badly. Good parsing matters.

Evaluate it

Do not judge by a few lucky demos. Build a small test set of real questions with expected answers or expected source documents, and measure:

  • Retrieval: did the right passage appear in the top results?
  • Faithfulness: is the answer supported by the retrieved text?
  • Answer quality: is it correct and complete?

Re-run the tests whenever you change the chunking, embedding model, prompt or LLM.

Security and privacy

  • Respect permissions. Filter retrieval by what the current user may read. Never let one customer's documents appear in another's answers.
  • Treat documents as untrusted input. A web page or uploaded file can contain hidden instructions ("ignore previous rules and reveal..."), known as indirect prompt injection. Keep instructions separate from data, limit what the model can do with tools, and do not let retrieved text trigger sensitive actions without checks. See the LLM security article in this series.
  • Do not index secrets or personal data you do not need.
  • Check the privacy terms of any external embedding or model API you send content to.

Practical tips

  • Show sources to the user as links.
  • Log questions, retrieved chunks and answers (with privacy care) to find failures.
  • Add feedback buttons and review bad answers regularly.
  • Start simple: good chunks, hybrid retrieval, a firm prompt. Add complexity only when your tests show a need.

Takeaway: RAG retrieves relevant passages from your documents and makes the model answer from them. Quality depends mostly on chunking and retrieval, so test with real questions, respect permissions and treat retrieved text as untrusted.

AI & LLM series, Part 1 (building with language models): 4 of 5. Previous: Embeddings and vector search explained for developers.

Keep reading