Building a RAG pipeline from scratch

By Ryze AI Team

Retrieval-augmented generation is the most reliable way to make an LLM useful on your own data without training anything. Instead of baking knowledge into the model through fine-tuning, you retrieve the relevant pieces of your data at query time and hand them to the model as context. The result is cheaper than fine-tuning, updatable when your data changes, and auditable, because you can see exactly which passages informed the answer.

The pieces

A RAG pipeline has five stages. Chunking splits your documents into pieces of roughly 200 to 500 tokens with overlap between them. Embedding encodes each chunk into a vector. Storage keeps those vectors somewhere queryable. Retrieval embeds the user's question and finds the nearest chunks. Generation prepends those chunks to the prompt and asks the model to answer. Each stage is simple; the quality comes from getting the details right, especially chunking and retrieval.

Chunking strategy

Chunking is where most pipelines silently fail. Fixed token counts are easy but cut sentences and facts in half. Prefer splitting on semantic boundaries such as headings, paragraphs, or sections, and only fall back to fixed sizes when the structure is poor. Keep an overlap of 10 to 20 percent between chunks so that a fact split across a boundary still appears whole in at least one chunk. Store metadata with every chunk: source document, page number, section title, and creation date. That metadata is what lets you cite sources later and filter by date or document type.

Chunk size involves a trade-off. Smaller chunks are more precise but may lack context. Larger chunks give the model more surrounding information but dilute the signal and consume prompt budget. 300 to 500 tokens is a reasonable default for prose; code and tables often need different handling.

Embeddings

Use a dedicated embedding model, not the LLM itself. Small models such as all-MiniLM-L6-v2 or BGE-small are fast, cheap, and good enough for most use cases. Normalize vectors so that cosine similarity reduces to a dot product, which makes retrieval faster. If your domain is specialized, consider a domain-tuned embedding model, but measure first; the gain is often smaller than expected.

Embed once at ingestion time and store the vectors. Re-embedding on every query is wasteful and slow. When you change embedding models, you must re-embed everything, so pick carefully and version your embeddings.

Retrieval quality

Top-k of three to five chunks is usually right. More chunks dilute the prompt with irrelevant context and push useful information out of the model's attention window. Fewer risks missing the answer. Add a reranker, such as a cross-encoder, when precision matters more than latency; the reranker scores each candidate chunk against the query and reorders them.

Hybrid search, combining keyword matching with vector similarity, beats pure vector search on exact terms, names, and identifiers. Many vector databases support this natively. If yours does not, run both and merge the results with reciprocal rank fusion.

Failure modes and mitigation

The common failures are: retrieval misses the relevant passage and the model hallucinates; the data is stale; chunk boundaries cut the answer in half; and the prompt is too long for the model to use all the context. Mitigate with citations so users can verify, an abstention instruction telling the model to say it does not know when the context lacks the answer, and evaluation on real questions from your users rather than synthetic ones. Log retrieved chunks alongside answers so you can diagnose misses after the fact.

Resources: Pinecone's RAG guide, LangChain, and LlamaIndex.