Create a RAG app with LangChain and Chroma

By Ryze AI Team

This tutorial builds a question-answering app over your own documents using LangChain for orchestration and Chroma as the vector store. By the end you will have a working pipeline you can point at any text.

Install dependencies

pip install langchain langchain-community chromadb sentence-transformers openai. Chroma runs embedded, so there is no separate server to start.

Load and chunk documents

from langchain_community.document_loaders import TextLoader, PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = TextLoader("data.txt")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)
print(len(chunks), "chunks")

RecursiveCharacterTextSplitter tries to split on paragraphs, then sentences, then characters, which keeps chunks semantically coherent. Tune chunk_size and chunk_overlap to your content; 500 and 50 are reasonable defaults for prose.

Embed and store

from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma

emb = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
db = Chroma.from_documents(chunks, emb, persist_directory="./chroma_db")
retriever = db.as_retriever(search_kwargs={"k": 4})

The embedding model runs locally, so no API key is needed for retrieval. Persisting to disk means you do not re-embed on every run.

Query and generate

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA

llm = ChatOpenAI(model="gpt-4o-mini")
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever, return_source_documents=True)
result = qa.invoke({"query": "What is the refund policy?"})
print(result["result"])
for d in result["source_documents"]:
    print("source:", d.metadata)

Improve quality

Add an abstention instruction to the prompt so the model says it does not know when the context lacks the answer. Increase k or add a reranker if recall is poor. Log retrieved chunks alongside answers to diagnose misses. And evaluate on real questions before shipping.

Common issues

If answers ignore the context, your prompt may not emphasize using it; add "Answer using only the provided context." If retrieval returns irrelevant chunks, check chunking quality and try a stronger embedding model. If Chroma is slow, ensure you are persisting and not rebuilding the index each run.

Docs: LangChain, Chroma.