How to Build a Local RAG Pipeline with Ollama and LangChain
Retrieval-augmented generation (RAG) lets you answer questions over your own documents by combining a retriever with a generator. Most hosted RAG tutorials send your documents and questions to a third-party API. This article shows how to build the same pipeline entirely on your own machine, using Ollama for embeddings and text generation, LangChain for orchestration, and Chroma for vector storage. Unless you configure remote endpoints, nothing leaves your computer.
By the end of this article you will have a runnable Python script that loads documents, splits them into chunks, embeds the chunks, stores the vectors, retrieves the most relevant chunks for a question, and generates an answer grounded in those chunks.
Prerequisites
- Ollama installed and running (
ollama serve). Follow the official installation instructions for your operating system. - Two Ollama models: an embedding model such as
nomic-embed-textand a generation model such asllama3.2. Other models work; change the names in the examples. - Python 3 and the packages
langchain-ollama,langchain-community, andchromadb.
How Local RAG Works
A local RAG pipeline runs in six stages:
- Load documents from files.
- Split documents into small chunks.
- Embed each chunk into a vector with the embedding model.
- Store the vectors in a vector database. This article uses Chroma.
- Embed the user's question and retrieve the most similar chunks.
- Feed the question together with the retrieved chunks to the generation model, which answers using only that context.
Use the same embedding model when indexing and when querying. Otherwise the vectors live in different spaces, and similarity search returns noise.
Step 1 — Start Ollama and Pull Models
Start the Ollama server:
ollama serve
Pull the models:
ollama pull nomic-embed-text
ollama pull llama3.2
These are examples of model families, not requirements. Any embedding model and chat model available in Ollama can substitute; adjust the model names in the code below.
Step 2 — Load and Split Documents
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
import pathlib
docs_dir = pathlib.Path("docs")
docs_dir.mkdir(exist_ok=True)
(docs_dir / "rag-intro.txt").write_text(
"Retrieval-augmented generation combines a retriever with a generator. "
"A local pipeline keeps documents and vectors on your own machine."
)
(docs_dir / "ollama-basics.txt").write_text(
"Ollama runs large language models locally on your computer. "
"It provides an HTTP API for chat and embeddings on port 11434."
)
loader = TextLoader(str(docs_dir / "rag-intro.txt"))
loader2 = TextLoader(str(docs_dir / "ollama-basics.txt"))
documents = loader.load() + loader2.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)
print(f"{len(chunks)} chunks ready for indexing")
Splitters break documents into overlapping chunks so related sentences stay together while each chunk stays small enough for effective similarity search.
Step 3 — Embed and Store in Chroma
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="./chroma_db",
)
vectorstore.persist()
print("Vectors stored in ./chroma_db")
OllamaEmbeddings calls Ollama's embedding endpoint for every chunk. persist_directory saves the index to disk so the vectors survive script restarts. Reuse the same directory and embedding model when you query later.
Step 4 — Retrieve Relevant Context
question = "How does a local RAG pipeline keep data private?"
retrieved = vectorstore.similarity_search(question, k=4)
context = "\n\n".join(doc.page_content for doc in retrieved)
print(context)
The question is embedded with the same model used for indexing. Chroma returns the k chunks whose vectors are closest to the question vector, typically by cosine similarity. These chunks become the evidence for the answer.
Step 5 — Generate the Answer
from langchain_ollama import OllamaLLM
from langchain_core.prompts import PromptTemplate
llm = OllamaLLM(model="llama3.2")
template = """Answer the question using only the provided context.
If the answer is not in the context, say so.
Context:
{context}
Question:
{question}
Answer:"""
prompt = PromptTemplate(template=template, input_variables=["context", "question"])
chain = prompt | llm
answer = chain.invoke({"context": context, "question": question})
print(answer)
The prompt tells the model to ground its answer in the retrieved context and to admit when the context is insufficient. This reduces invented answers. Swap llama3.2 for any chat model you have pulled in Ollama.
Common Mistakes
- Using different embedding models for indexing and querying.
- Chunks that are too large (diluted relevance) or too small (lost context), and chunks without overlap that split sentences across boundaries.
- Forgetting
persist_directory, so the vectors vanish when the script ends. - Expecting small local models to match hosted flagship quality. Adjust expectations, not just prompts.
- Pasting API keys, passwords, or personal data into documents the pipeline indexes.
Trade-Offs
Local RAG keeps documents private and removes per-request API fees, but it runs on your hardware. Available RAM and CPU or GPU determine which models are practical and how fast answers arrive. Generation quality depends on the models you pull. You are also responsible for updates and maintenance. Hosted APIs trade privacy and cost for convenience and consistent model quality.
Next Actions
Run the pipeline with your own documents. Tune chunk_size and chunk_overlap, change k, try different Ollama models, add reranking, or swap Chroma for pgvector, Qdrant, or Milvus as the corpus grows.
FAQ
Do I need an API key? No. Ollama runs embedding and generation models locally, so no third-party API key is required unless you connect a remote endpoint.
Which models should I use?
The examples use nomic-embed-text for embeddings and llama3.2 for generation. They work well as starting points; any Ollama embedding and chat model can replace them.
Can I use another vector store? Yes. Chroma is used here for simplicity. LangChain supports pgvector, Qdrant, Milvus, Weaviate, and others with the same indexing and retrieval pattern.
Is my data private? Yes, when Ollama and Chroma run on your own machine. Documents and vectors stay on your host unless you explicitly configure remote services.
Do I need a GPU? No. Ollama runs on CPU. A GPU only speeds up embedding and generation.