What is RAG? A Practical Introduction for Engineers

After a long break, I am writing again on TechMyTalk — this time about the technology I work on every day: Retrieval-Augmented Generation, or RAG.

What is RAG?

A large language model (LLM) is good at language, but it does not know your company’s documents, your tickets, or your runbooks. RAG fixes that. Instead of asking the model to answer from memory alone, you first retrieve the relevant documents, then give them to the model along with the question. The model answers from your documents, not from what it happened to learn in training.

In short: retrieve first, then generate.

The three steps

1. Index — Split your documents into chunks, turn each chunk into a vector embedding, and store them in a vector database such as Elasticsearch.
2. Retrieve — Turn the user’s question into the same kind of embedding, search for the closest chunks, and rank them. A hybrid search — keyword (BM25) plus vector (kNN) — usually works better than either alone.
3. Generate — Send the question plus the top retrieved chunks to the LLM, with a simple instruction: answer only from these chunks, and say you do not know if the answer is not there.

Why it matters

• Answers are grounded in your own documents, so hallucinations go down.
• You can update the knowledge by updating the documents — no retraining.
• You can show the source of each answer, so people can check it.

Where it gets hard in the enterprise

The basic idea is simple. Making it work at enterprise scale is not. The hard parts are: keeping access control (a user must only see documents they are allowed to see), handling many languages and many document types, keeping the index fresh, and knowing whether the retrieved chunks actually answer the question.

I will write about those in the next posts — including GraphRAG, where a knowledge graph is built over the documents to answer questions that plain vector search cannot.

If you are building RAG, start simple: one document set, hybrid search, and a strict prompt. Measure whether the retrieved chunks answer the question, before you add anything more complex.