What Is RAG (Retrieval-Augmented Generation)

RAG (retrieval-augmented generation) is a pattern where you search your own documents for the passages most relevant to a user's question, then hand only those passages to an LLM to write the answer. The model answers from your data instead of its training memory, which cuts hallucinations and keeps answers current.

How Does the Retrieve-Then-Generate Flow Work?

Offline, you split documents into chunks and store an embedding for each one; at query time you embed the question, fetch the closest chunks, and prompt the LLM with question plus chunks.

# Offline (once per document update)
chunks = split_documents(docs)            # 1. chunk
vectors = embed(chunks)                   # 2. embed
index.store(chunks, vectors)              # 3. index

# At query time
q_vec = embed(question)                   # 4. embed the question
top_chunks = index.search(q_vec, k=5)     # 5. retrieve nearest chunks
answer = llm(f"Answer using only this context:\n{top_chunks}\n\nQ: {question}")

Why Not Just Stuff Everything Into the Context Window?

Even with million-token context windows, stuffing is slower, costs more per query, and models reliably pay less attention to material buried in the middle of a huge prompt. Retrieval sends only the few thousand relevant tokens, so answers are cheaper, faster, and better grounded — and your corpus can be far bigger than any context window.

How Should You Chunk Documents?

Chunking is the highest-leverage design choice in RAG: chunks must be small enough to be precise but large enough to be understood alone. A common starting point is 300-800 tokens with 10-15% overlap, splitting on natural boundaries (headings, paragraphs) rather than a blind character count, and prepending the document title to each chunk so it carries its own context.

How Do You Evaluate a RAG System?

Evaluate the two stages separately, because a wrong answer can come from bad retrieval or bad generation.

• Retrieval: for a set of test questions with known source passages, measure recall@k — did the right chunk appear in the top k results?
• Generation: check faithfulness (is every claim supported by the retrieved chunks?) and answer relevance, using human review or an LLM judge.
• End to end: keep a fixed question set with reference answers and rerun it after every change to chunking, embedding model, or prompt.

Common Pitfalls

• Debugging the prompt when retrieval is broken: if the right chunk never gets retrieved, no prompt can save you. Log and inspect retrieved chunks first.
• Chunks that orphan their context: a table row or pronoun-heavy paragraph is meaningless alone. Include headings and titles in each chunk.
• Pure vector search: embeddings miss exact identifiers like SKUs and error codes. Hybrid search (vector + keyword) fixes this cheaply.
• No instruction for missing answers: tell the model to say "not found in the provided context" instead of guessing.